{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wide and Deep on TensorFlow (notebook style)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2016 Google Inc. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n",
    " http://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "This notebook uses the tf.learn API in TensorFlow to answer a yes/no question. This is called a binary classification problem: Given census data about a person such as age, gender, education and occupation (the features), we will try to predict whether or not the person earns more than 50,000 dollars a year (the target label). \n",
    "\n",
    "Given an individual's information our model will output a number between 0 and 1, which can be interpreted as the model's certainty that the individual has an annual income of over 50,000 dollars, (1=True, 0=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imports and constants\n",
    "First we'll import our libraries and set up some strings for column names. We also print out the version of TensorFlow we are running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using TensorFlow version 1.2.1\n",
      "\n",
      "Feature columns are:  ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13', 'C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26'] \n",
      "\n",
      "Columns and data as a dict:  {'I1': 0, 'I2': 127, 'I3': 1, 'I4': 3, 'I5': 1683, 'I6': 19, 'I7': 26, 'I8': 17, 'I9': 475, 'I10': 0, 'I11': 9, 'I12': 0, 'I13': 3, 'C1': '05db9164', 'C2': '8947f767', 'C3': '11c9d79e', 'C4': '52a787c8', 'C5': '4cf72387', 'C6': 'fbad5c96', 'C7': '18671b18', 'C8': '0b153874', 'C9': 'a73ee510', 'C10': 'ceb10289', 'C11': '77212bd7', 'C12': '79507c6b', 'C13': '7203f04e', 'C14': '07d13a8f', 'C15': '2c14c412', 'C16': '49013ffe', 'C17': '8efede7f', 'C18': 'bd17c3da', 'C19': 'f6a3e43b', 'C20': 'a458ea53', 'C21': '35cd95c9', 'C22': 'ad3062eb', 'C23': 'c7dc6720', 'C24': '3fdb382b', 'C25': '010f6491', 'C26': '49d68486'} \n",
      "\n"
     ]
    }
   ],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import time\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.INFO) # Set to INFO for tracking training, default is WARN. ERROR for least messages\n",
    "\n",
    "print(\"Using TensorFlow version %s\\n\" % (tf.__version__))\n",
    "\n",
    "\n",
    "CONTINUOUS_COLUMNS =  [\"I\"+str(i) for i in range(1,14)] # 1-13 inclusive\n",
    "CATEGORICAL_COLUMNS = [\"C\"+str(i) for i in range(1,27)] # 1-26 inclusive\n",
    "LABEL_COLUMN = [\"clicked\"]\n",
    "\n",
    "TRAIN_DATA_COLUMNS = LABEL_COLUMN + CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS\n",
    "# TEST_DATA_COLUMNS = CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS\n",
    "\n",
    "FEATURE_COLUMNS = CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS\n",
    "\n",
    "print('Feature columns are: ', FEATURE_COLUMNS, '\\n')\n",
    "\n",
    " # label is 1\n",
    "sample = [ 0 , 2, 11, 5, 10262, 34, 2, 4, 5,0 , 1,0 , 5, \"be589b51\", \"287130e0\", \"cd7a7a22\", \"fb7334df\", \"25c83c98\",\"0\" , \"6cdb3998\", \"361384ce\", \"a73ee510\", \"3ff10fb2\", \"5874c9c9\", \"976cbd4c\", \"740c210d\", \"1adce6ef\", \"310d155b\", \"07eb8110\", \"07c540c4\", \"891589e7\", \"18259a83\", \"a458ea53\", \"a0ab60ca\",\"0\" , \"32c7478e\", \"a052b1ed\", \"9b3e8820\", \"8967c0d2\"]\n",
    "\n",
    "# label is 1\n",
    "sample = [ 0, 127, 1, 3, 1683, 19, 26, 17, 475, 0, 9, 0, 3, \"05db9164\", \"8947f767\", \"11c9d79e\", \"52a787c8\", \"4cf72387\", \"fbad5c96\", \"18671b18\", \"0b153874\", \"a73ee510\", \"ceb10289\", \"77212bd7\", \"79507c6b\", \"7203f04e\", \"07d13a8f\", \"2c14c412\", \"49013ffe\", \"8efede7f\", \"bd17c3da\", \"f6a3e43b\", \"a458ea53\", \"35cd95c9\", \"ad3062eb\", \"c7dc6720\", \"3fdb382b\", \"010f6491\", \"49d68486\"]\n",
    "\n",
    "print('Columns and data as a dict: ', dict(zip(FEATURE_COLUMNS, sample)), '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Input file parsing\n",
    "\n",
    "This section puts the file into a `Reader` which reads from the file one batch at a time. \n",
    "\n",
    "We set up the Tensors to be a dictionary of features mapping from their string name to the tensor value.\n",
    "\n",
    "Note that the `_input_fn()` function is wrapped, enabling it to be used for different files.\n",
    "\n",
    "NOTE: This reads from the input file directly via TensorFlow, rather than using an intermediate tool such as pandas to load the entire dataset into memory first. This is done to enable the system to scale to large inputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More about input functions\n",
    "\n",
    "The input function is how we will feed the input data into the model during training and evaluation. \n",
    "The structure that must be returned is a pair, where the first element is a dict of the column names (features) mapped to a tensor of values, and the 2nd element is a tensor of values representing the answers (labels). Recall that a tensor is just a general term for an n-dimensional array.\n",
    "\n",
    "This could be represented as: `map(column_name => [Tensor of values]) , [Tensor of labels])`\n",
    "\n",
    "More concretely, for this particular dataset, something like this:\n",
    "\n",
    "    { \n",
    "      'age':            [ 39, 50, 38, 53, 28, … ], \n",
    "      'marital_status': [ 'Married-civ-spouse', 'Never-married', 'Widowed', 'Widowed' … ],\n",
    "       ...\n",
    "      'gender':           ['Male', 'Female', 'Male', 'Male', 'Female',, … ], \n",
    "    } , \n",
    "    [ 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1]\n",
    "    \n",
    "Additionally, we define which columns of the input data we will treat as categorical vs continuous, using the global `CATEGORICAL_COLUMNS`.\n",
    "\n",
    "You can try different values for `BATCH_SIZE` to see how they impact your results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### High-level structure of input functions for CSV-style data\n",
    "1. Queue file(s)\n",
    "2. Read a batch of data from the next file\n",
    "3. Create record defaults, generally 0 for continuous values, and \"\" for categorical. You can use named types if you prefer\n",
    "4. Decode the CSV and restructure it to be appropriate for the graph's input format\n",
    "    * `zip()` column headers with the data\n",
    "    * `pop()` off the label column(s)\n",
    "    * Remove/pop any unneeded column(s)\n",
    "    * Run `tf.expand_dims()` on categorical columns\n",
    "    5. Return the pair: `(feature_dict, label_array)`\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input function configured\n"
     ]
    }
   ],
   "source": [
    "BATCH_SIZE = 400\n",
    "\n",
    "def generate_input_fn(filename, batch_size=BATCH_SIZE):\n",
    "    def _input_fn():\n",
    "        #将文件名列表交给tf.train.string_input_producer函数，\n",
    "        #string_input_producer会生成一个先入先出的队列， 文件阅读器会需要它来读取数据\n",
    "        \n",
    "        filename_queue = tf.train.string_input_producer([filename])\n",
    "        \n",
    "        # Create an textline reader\n",
    "        reader = tf.TextLineReader()\n",
    "        # Reads out batch_size number of lines\n",
    "        key, value = reader.read_up_to(filename_queue, num_records=batch_size)\n",
    "        \n",
    "        # 1 int label, 13 ints, 26 strings\n",
    "        cont_defaults = [ [0] for i in range(1,14) ]\n",
    "        cate_defaults = [ [\" \"] for i in range(1,27) ]\n",
    "        label_defaults = [ [0] ]\n",
    "        column_headers = TRAIN_DATA_COLUMNS\n",
    "        # The label is the first column of the data.\n",
    "        record_defaults = label_defaults + cont_defaults + cate_defaults\n",
    "\n",
    "        # Decode CSV data that was just read out. \n",
    "        # Note that this does NOT return a dict, \n",
    "        # so we will need to zip it up with our headers\n",
    "        #deocde_csv interprets return values as tensor type .\n",
    "        \n",
    "        #Convert CSV records to tensors. Each column maps to one tensor\n",
    "        columns = tf.decode_csv(\n",
    "            value, record_defaults=record_defaults)\n",
    "        \n",
    "        # all_columns is a dictionary that maps from column names to tensors of the data.\n",
    "        all_columns = dict(zip(column_headers, columns))\n",
    "        \n",
    "        # Pop and save our labels \n",
    "        # dict.pop() returns the popped array of values; exactly what we need!\n",
    "        labels = all_columns.pop(LABEL_COLUMN[0])\n",
    "        \n",
    "        # the remaining columns are our features\n",
    "        features = all_columns \n",
    "        print(\"origin_features:%s\"%features)\n",
    "        # Sparse categorical features must be represented with an additional dimension. \n",
    "        # There is no additional work needed for the Continuous columns; they are the unaltered columns.\n",
    "        # See docs for tf.SparseTensor for more info\n",
    "        for feature_name in CATEGORICAL_COLUMNS:\n",
    "            features[feature_name] = tf.expand_dims(features[feature_name], -1)\n",
    "        print(\"expand_features:%s\"%features)\n",
    "        \n",
    "        return features, labels\n",
    "\n",
    "    return _input_fn\n",
    "\n",
    "print('input function configured')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Feature Columns\n",
    "This section configures the model with the information about the model. There are many parameters here to experiment with to see how they affect the accuracy.\n",
    "\n",
    "This is the bulk of the time and energy that is often spent on making a machine learning model work, called *feature selection* or *feature engineering*. We choose the features (columns) we will use for training, and apply any additional transformations to them as needed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sparse Columns\n",
    "First we build the sparse columns.\n",
    "\n",
    "Use `sparse_column_with_keys()` for columns that we know all possible values for.\n",
    "\n",
    "Use `sparse_column_with_hash_bucket()` for columns that we want the the library to automatically map values for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wide_col:[_SparseColumnHashed(column_name='C1', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C2', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C3', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C4', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C5', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C6', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C7', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C8', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C9', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C10', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C11', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C12', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C13', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C14', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C15', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C16', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C17', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C18', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C19', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C20', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C21', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C22', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C23', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C24', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C25', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C26', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string)]\n",
      "Wide/Sparse columns configured\n"
     ]
    }
   ],
   "source": [
    "# Sparse base columns.\n",
    "# C1 = tf.contrib.layers.sparse_column_with_hash_bucket('C1', hash_bucket_size=1000)\n",
    "# C2 = tf.contrib.layers.sparse_column_with_hash_bucket('C2', hash_bucket_size=1000)\n",
    "# C3 = tf.contrib.layers.sparse_column_with_hash_bucket('C3', hash_bucket_size=1000)\n",
    "# ...\n",
    "# Cn = tf.contrib.layers.sparse_column_with_hash_bucket('Cn', hash_bucket_size=1000)\n",
    "# wide_columns = [C1, C2, C3, ... , Cn]\n",
    "\n",
    "wide_columns = []\n",
    "for name in CATEGORICAL_COLUMNS:\n",
    "    wide_columns.append(tf.contrib.layers.sparse_column_with_hash_bucket(\n",
    "            name, hash_bucket_size=1000))\n",
    "print(\"Wide_col:%s\"%wide_columns)\n",
    "print('Wide/Sparse columns configured')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Continuous columns\n",
    "Second, configure the real-valued columns using `real_valued_column()`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deep_col:[_RealValuedColumn(column_name='I1', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I2', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I3', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I4', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I5', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I6', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I7', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I8', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I9', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I10', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I11', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I12', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I13', dimension=1, default_value=None, dtype=tf.float32, normalizer=None)]\n",
      "deep/continuous columns configured\n"
     ]
    }
   ],
   "source": [
    "# Continuous base columns.\n",
    "# I1 = tf.contrib.layers.real_valued_column(\"I1\")\n",
    "# I2 = tf.contrib.layers.real_valued_column(\"I2\")\n",
    "# I3 = tf.contrib.layers.real_valued_column(\"I3\")\n",
    "# ...\n",
    "# In = tf.contrib.layers.real_valued_column(\"In\")\n",
    "# deep_columns = [I1, I2, I3, ... , In]\n",
    "\n",
    "deep_columns = []\n",
    "for name in CONTINUOUS_COLUMNS:\n",
    "    deep_columns.append(tf.contrib.layers.real_valued_column(name))\n",
    "print(\"deep_col:%s\" %deep_columns)\n",
    "print('deep/continuous columns configured')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformations\n",
    "Now for the interesting stuff. We will employ a couple of techniques to get even more out of the data.\n",
    " \n",
    "* **bucketizing** turns what would have otherwise been a continuous feature into a categorical one. \n",
    "* **feature crossing** allows us to compute a model weight for specific pairings across columns, rather than learning them as independently. This essentially encodes related columns together, for situations where having 2 (or more) columns being certain values is meaningful. \n",
    "\n",
    "Only categorical features can be crossed. This is one reason why age has been bucketized.\n",
    "\n",
    "For example, crossing education and occupation would enable the model to learn about: \n",
    "\n",
    "    education=\"Bachelors\" AND occupation=\"Exec-managerial\"\n",
    "\n",
    "or perhaps \n",
    "\n",
    "    education=\"Bachelors\" AND occupation=\"Craft-repair\"\n",
    "\n",
    "We do a few combined features (feature crosses) here. \n",
    "\n",
    "Add your own, based on your intuitions about the dataset, to try to improve on the model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# No known Transformations. Can add some if desired. \n",
    "# Examples from other datasets are shown below.\n",
    "\n",
    "# age_buckets = tf.contrib.layers.bucketized_column(age,\n",
    "#             boundaries=[ 18, 25, 30, 35, 40, 45, 50, 55, 60, 65 ])\n",
    "# education_occupation = tf.contrib.layers.crossed_column([education, occupation], \n",
    "#                                                         hash_bucket_size=int(1e4))\n",
    "# age_race_occupation = tf.contrib.layers.crossed_column([age_buckets, race, occupation], \n",
    "#                                                        hash_bucket_size=int(1e6))\n",
    "# country_occupation = tf.contrib.layers.crossed_column([native_country, occupation], \n",
    "#                                                       hash_bucket_size=int(1e4))\n",
    "\n",
    "print('Transformations complete')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Group feature columns into 2 objects\n",
    "\n",
    "The wide columns are the sparse, categorical columns that we specified, as well as our hashed, bucket, and feature crossed columns. \n",
    "\n",
    "The deep columns are composed of embedded categorical columns along with the continuous real-valued columns. **Column embeddings** transform a sparse, categorical tensor into a low-dimensional and dense real-valued vector. The embedding values are also trained along with the rest of the model. For more information about embeddings, see the TensorFlow tutorial on [Vector Representations Words](https://www.tensorflow.org/tutorials/word2vec/), or [Word Embedding](https://en.wikipedia.org/wiki/Word_embedding) on Wikipedia.\n",
    "\n",
    "The higher the dimension of the embedding is, the more degrees of freedom the model will have to learn the representations of the features. We are starting with an 8-dimension embedding for simplicity, but later you can come back and increase the dimensionality if you wish.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "WARNING:tensorflow:The default stddev value of initializer will change from \"1/sqrt(vocab_size)\" to \"1/sqrt(dimension)\" after 2017/02/25.\n",
      "deep_col:[_RealValuedColumn(column_name='I1', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I2', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I3', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I4', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I5', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I6', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I7', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I8', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I9', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I10', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I11', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I12', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _RealValuedColumn(column_name='I13', dimension=1, default_value=None, dtype=tf.float32, normalizer=None), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C1', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394320>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C2', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394390>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C3', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x00000217673944A8>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C4', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394358>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C5', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767275208>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C6', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x00000217604ADC88>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C7', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394438>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C8', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x00000217673945C0>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C9', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x00000217673944E0>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C10', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x00000217673945F8>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C11', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394400>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C12', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394518>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C13', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x00000217673946A0>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C14', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394630>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C15', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x00000217673946D8>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C16', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394748>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C17', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394470>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C18', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x00000217673947F0>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C19', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394588>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C20', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394668>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C21', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394710>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C22', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394780>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C23', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x00000217673948D0>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C24', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394908>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C25', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394828>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True), _EmbeddingColumn(sparse_id_column=_SparseColumnHashed(column_name='C26', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), dimension=8, combiner='mean', initializer=<tensorflow.python.ops.init_ops.TruncatedNormal object at 0x0000021767394898>, ckpt_to_load_from=None, tensor_name_in_ckpt=None, shared_embedding_name=None, shared_vocab_size=None, max_norm=None, trainable=True)]\n",
      "wide_col:[_SparseColumnHashed(column_name='C1', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C2', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C3', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C4', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C5', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C6', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C7', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C8', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C9', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C10', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C11', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C12', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C13', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C14', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C15', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C16', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C17', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C18', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C19', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C20', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C21', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C22', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C23', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C24', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C25', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string), _SparseColumnHashed(column_name='C26', is_integerized=False, bucket_size=1000, lookup_config=None, combiner='sum', dtype=tf.string)]\n",
      "wide and deep columns configured\n"
     ]
    }
   ],
   "source": [
    "# Wide columns and deep columns.\n",
    "# wide_columns = [gender, race, native_country,\n",
    "#       education, occupation, workclass,\n",
    "#       marital_status, relationship,\n",
    "#       age_buckets, education_occupation,\n",
    "#       age_race_occupation, country_occupation]\n",
    "\n",
    "# deep_columns = [\n",
    "#   tf.contrib.layers.embedding_column(workclass, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(education, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(marital_status, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(gender, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(relationship, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(race, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(native_country, dimension=8),\n",
    "#   tf.contrib.layers.embedding_column(occupation, dimension=8),\n",
    "#   age,\n",
    "#   education_num,\n",
    "#   capital_gain,\n",
    "#   capital_loss,\n",
    "#   hours_per_week,\n",
    "# ]\n",
    "\n",
    "# Embeddings for wide columns into deep columns\n",
    "for col in wide_columns:\n",
    "    deep_columns.append(tf.contrib.layers.embedding_column(col, \n",
    "                                                           dimension=8))\n",
    "print(\"deep_col:%s\" %deep_columns)\n",
    "print(\"wide_col:%s\" %wide_columns)\n",
    "print('wide and deep columns configured')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create the model\n",
    "\n",
    "You can train either a \"wide\" model, a \"deep\" model, or a \"wide and deep\" model, using the classifiers below. Try each one and see what kind of results you get.\n",
    "\n",
    "* **Wide**: Linear Classifier\n",
    "* **Deep**: Deep Neural Net Classifier\n",
    "* **Wide & Deep**: Combined Linear and Deep Classifier\n",
    "\n",
    "The `hidden_units` or `dnn_hidden_units` argument is to specify the size of each layer of the deep portion of the network. For example, `[12, 20, 15]` would create a network with the first layer of size 12, the second layer of size 20, and a third layer of size 15."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model directory = models/model_WIDE_AND_DEEP_1556769598\n",
      "WARNING:tensorflow:From <ipython-input-6-6aeab337b09f>:40: calling DNNLinearCombinedClassifier.__init__ (from tensorflow.contrib.learn.python.learn.estimators.dnn_linear_combined) with fix_global_step_increment_bug=False is deprecated and will be removed after 2017-04-15.\n",
      "Instructions for updating:\n",
      "Please set fix_global_step_increment_bug=True and update training steps in your pipeline. See pydoc for details.\n",
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x00000217673AD7B8>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_session_config': None, '_save_checkpoints_steps': 100, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'models/model_WIDE_AND_DEEP_1556769598'}\n",
      "estimator built\n"
     ]
    }
   ],
   "source": [
    "def create_model_dir(model_type):\n",
    "    # Returns something like models/model_WIDE_AND_DEEP_1493043407\n",
    "    return 'models/model_' + model_type + '_' + str(int(time.time()))\n",
    "\n",
    "# Specify the desired model_dir \n",
    "def get_model(model_type, model_dir):\n",
    "    print(\"Model directory = %s\" % model_dir)\n",
    "    \n",
    "    # There are more options here than shown here. \n",
    "    # We are using this to show additional checkpointing for illustrative purposes.\n",
    "    # In a real system with far more samples, you would \n",
    "    #     likely choose to save checkpoints less frequently.\n",
    "    runconfig = tf.contrib.learn.RunConfig(\n",
    "        save_checkpoints_secs=None,\n",
    "        save_checkpoints_steps = 100,\n",
    "    )\n",
    "    \n",
    "    m = None\n",
    "    \n",
    "    # Linear Classifier\n",
    "    if model_type == 'WIDE':\n",
    "        m = tf.contrib.learn.LinearClassifier(\n",
    "            model_dir=model_dir, \n",
    "            feature_columns=wide_columns)\n",
    "\n",
    "    # Deep Neural Net Classifier\n",
    "    if model_type == 'DEEP':\n",
    "        m = tf.contrib.learn.DNNClassifier(\n",
    "            model_dir=model_dir,\n",
    "            feature_columns=deep_columns,\n",
    "            hidden_units=[100, 50, 25])\n",
    "\n",
    "    # Combined Linear and Deep Classifier\n",
    "    if model_type == 'WIDE_AND_DEEP':\n",
    "        m = tf.contrib.learn.DNNLinearCombinedClassifier(\n",
    "            model_dir=model_dir,\n",
    "            linear_feature_columns=wide_columns,\n",
    "            dnn_feature_columns=deep_columns,\n",
    "            dnn_hidden_units=[100, 70, 50, 25],\n",
    "            config=runconfig)\n",
    "        \n",
    "    print('estimator built')\n",
    "    \n",
    "    return m\n",
    "    \n",
    "\n",
    "MODEL_TYPE = 'WIDE_AND_DEEP'\n",
    "model_dir = create_model_dir(model_type=MODEL_TYPE)\n",
    "m = get_model(model_type=MODEL_TYPE, model_dir=model_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Showing that canned estimators return an instance of 'Evaluable'\n",
    "\n",
    "from tensorflow.contrib.learn.python.learn import evaluable\n",
    "isinstance(m, evaluable.Evaluable)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fit the model (train it)\n",
    "\n",
    "Run `fit()` to train the model. You can experiment with the `train_steps` and `BATCH_SIZE` parameters.\n",
    "\n",
    "This can take some time, depending on the values chosen for `train_steps` and `BATCH_SIZE`.\n",
    "\n",
    "Our datafile is hosted on Google Cloud Storage; the reader we created at the beginning knows how to read from it.\n",
    "\n",
    "If you don't want to download a new copy of the dataset each time your script runs, you can download it locally using \n",
    "\n",
    "    gsutil cp gs://dataset-uploader/criteo-kaggle/medium_version/train.csv .\n",
    "    gsutil cp gs://dataset-uploader/criteo-kaggle/medium_version/eval.csv .\n",
    "    \n",
    "If you want to download it manually, use\n",
    "\n",
    "- http://storageapis.google.com/dataset-uploader/criteo-kaggle/medium_version/eval.csv\n",
    "- http://storageapis.google.com/dataset-uploader/criteo-kaggle/medium_version/train.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Use the cloud or local depending on your preference\n",
    "\n",
    "# CLOUD\n",
    "#train_file = \"gs://dataset-uploader/criteo-kaggle/medium_version/train.csv\"\n",
    "#eval_file  = \"gs://dataset-uploader/criteo-kaggle/medium_version/eval.csv\"\n",
    "\n",
    "# LOCAL. Update these paths as appropriate\n",
    "train_file = \"data_files/train.csv\"\n",
    "eval_file  = \"data_files/eval.csv\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "origin_features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "expand_features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'ExpandDims:0' shape=(?, 1) dtype=string>, 'C2': <tf.Tensor 'ExpandDims_1:0' shape=(?, 1) dtype=string>, 'C3': <tf.Tensor 'ExpandDims_2:0' shape=(?, 1) dtype=string>, 'C4': <tf.Tensor 'ExpandDims_3:0' shape=(?, 1) dtype=string>, 'C5': <tf.Tensor 'ExpandDims_4:0' shape=(?, 1) dtype=string>, 'C6': <tf.Tensor 'ExpandDims_5:0' shape=(?, 1) dtype=string>, 'C7': <tf.Tensor 'ExpandDims_6:0' shape=(?, 1) dtype=string>, 'C8': <tf.Tensor 'ExpandDims_7:0' shape=(?, 1) dtype=string>, 'C9': <tf.Tensor 'ExpandDims_8:0' shape=(?, 1) dtype=string>, 'C10': <tf.Tensor 'ExpandDims_9:0' shape=(?, 1) dtype=string>, 'C11': <tf.Tensor 'ExpandDims_10:0' shape=(?, 1) dtype=string>, 'C12': <tf.Tensor 'ExpandDims_11:0' shape=(?, 1) dtype=string>, 'C13': <tf.Tensor 'ExpandDims_12:0' shape=(?, 1) dtype=string>, 'C14': <tf.Tensor 'ExpandDims_13:0' shape=(?, 1) dtype=string>, 'C15': <tf.Tensor 'ExpandDims_14:0' shape=(?, 1) dtype=string>, 'C16': <tf.Tensor 'ExpandDims_15:0' shape=(?, 1) dtype=string>, 'C17': <tf.Tensor 'ExpandDims_16:0' shape=(?, 1) dtype=string>, 'C18': <tf.Tensor 'ExpandDims_17:0' shape=(?, 1) dtype=string>, 'C19': <tf.Tensor 'ExpandDims_18:0' shape=(?, 1) dtype=string>, 'C20': <tf.Tensor 'ExpandDims_19:0' shape=(?, 1) dtype=string>, 'C21': <tf.Tensor 'ExpandDims_20:0' shape=(?, 1) dtype=string>, 'C22': <tf.Tensor 'ExpandDims_21:0' shape=(?, 1) dtype=string>, 'C23': <tf.Tensor 'ExpandDims_22:0' shape=(?, 1) dtype=string>, 'C24': <tf.Tensor 'ExpandDims_23:0' shape=(?, 1) dtype=string>, 'C25': <tf.Tensor 'ExpandDims_24:0' shape=(?, 1) dtype=string>, 'C26': <tf.Tensor 'ExpandDims_25:0' shape=(?, 1) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 2 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:loss = 64.2209, step = 2\n",
      "INFO:tensorflow:Saving checkpoints for 102 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 15.471\n",
      "INFO:tensorflow:Saving checkpoints for 202 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.3595\n",
      "INFO:tensorflow:loss = 0.461707, step = 202 (12.571 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 302 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.7952\n",
      "INFO:tensorflow:Saving checkpoints for 402 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.2639\n",
      "INFO:tensorflow:loss = 0.441572, step = 402 (12.103 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 502 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.2508\n",
      "INFO:tensorflow:Saving checkpoints for 602 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.3401\n",
      "INFO:tensorflow:loss = 0.42619, step = 602 (12.273 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 702 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 17.3173\n",
      "INFO:tensorflow:Saving checkpoints for 802 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.6391\n",
      "INFO:tensorflow:loss = 0.41275, step = 802 (11.785 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 902 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.689\n",
      "INFO:tensorflow:Saving checkpoints for 1002 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.9944\n",
      "INFO:tensorflow:loss = 0.400731, step = 1002 (11.877 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1102 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.5567\n",
      "INFO:tensorflow:Saving checkpoints for 1202 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 17.1749\n",
      "INFO:tensorflow:loss = 0.391054, step = 1202 (11.861 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1302 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.2035\n",
      "INFO:tensorflow:Saving checkpoints for 1402 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.2009\n",
      "INFO:tensorflow:loss = 0.382504, step = 1402 (12.344 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1502 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 17.5078\n",
      "INFO:tensorflow:Saving checkpoints for 1602 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.0839\n",
      "INFO:tensorflow:loss = 0.37483, step = 1602 (11.930 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1702 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 15.5961\n",
      "INFO:tensorflow:Saving checkpoints for 1802 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 16.5978\n",
      "INFO:tensorflow:loss = 0.367093, step = 1802 (12.436 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1902 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 15.7159\n",
      "INFO:tensorflow:Saving checkpoints for 2002 into models/model_WIDE_AND_DEEP_1556769598\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 17.6496\n",
      "INFO:tensorflow:loss = 0.360448, step = 2002 (12.029 sec)\n",
      "INFO:tensorflow:Loss for final step: 0.360448.\n",
      "fit done\n",
      "Wall time: 2min 43s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# This can be found with\n",
    "# wc -l train.csv\n",
    "train_sample_size = 800000\n",
    "train_steps = train_sample_size/BATCH_SIZE # 8000/40 = 200\n",
    "\n",
    "m.fit(input_fn=generate_input_fn(train_file, BATCH_SIZE), steps=train_steps)\n",
    "\n",
    "print('fit done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluate the accuracy of the model\n",
    "Let's see how the model did. We will evaluate all the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "origin_features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "expand_features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'ExpandDims:0' shape=(?, 1) dtype=string>, 'C2': <tf.Tensor 'ExpandDims_1:0' shape=(?, 1) dtype=string>, 'C3': <tf.Tensor 'ExpandDims_2:0' shape=(?, 1) dtype=string>, 'C4': <tf.Tensor 'ExpandDims_3:0' shape=(?, 1) dtype=string>, 'C5': <tf.Tensor 'ExpandDims_4:0' shape=(?, 1) dtype=string>, 'C6': <tf.Tensor 'ExpandDims_5:0' shape=(?, 1) dtype=string>, 'C7': <tf.Tensor 'ExpandDims_6:0' shape=(?, 1) dtype=string>, 'C8': <tf.Tensor 'ExpandDims_7:0' shape=(?, 1) dtype=string>, 'C9': <tf.Tensor 'ExpandDims_8:0' shape=(?, 1) dtype=string>, 'C10': <tf.Tensor 'ExpandDims_9:0' shape=(?, 1) dtype=string>, 'C11': <tf.Tensor 'ExpandDims_10:0' shape=(?, 1) dtype=string>, 'C12': <tf.Tensor 'ExpandDims_11:0' shape=(?, 1) dtype=string>, 'C13': <tf.Tensor 'ExpandDims_12:0' shape=(?, 1) dtype=string>, 'C14': <tf.Tensor 'ExpandDims_13:0' shape=(?, 1) dtype=string>, 'C15': <tf.Tensor 'ExpandDims_14:0' shape=(?, 1) dtype=string>, 'C16': <tf.Tensor 'ExpandDims_15:0' shape=(?, 1) dtype=string>, 'C17': <tf.Tensor 'ExpandDims_16:0' shape=(?, 1) dtype=string>, 'C18': <tf.Tensor 'ExpandDims_17:0' shape=(?, 1) dtype=string>, 'C19': <tf.Tensor 'ExpandDims_18:0' shape=(?, 1) dtype=string>, 'C20': <tf.Tensor 'ExpandDims_19:0' shape=(?, 1) dtype=string>, 'C21': <tf.Tensor 'ExpandDims_20:0' shape=(?, 1) dtype=string>, 'C22': <tf.Tensor 'ExpandDims_21:0' shape=(?, 1) dtype=string>, 'C23': <tf.Tensor 'ExpandDims_22:0' shape=(?, 1) dtype=string>, 'C24': <tf.Tensor 'ExpandDims_23:0' shape=(?, 1) dtype=string>, 'C25': <tf.Tensor 'ExpandDims_24:0' shape=(?, 1) dtype=string>, 'C26': <tf.Tensor 'ExpandDims_25:0' shape=(?, 1) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-02-04:09:28\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556769598\\model.ckpt-2002\n",
      "INFO:tensorflow:Evaluation [1/500]\n",
      "INFO:tensorflow:Evaluation [2/500]\n",
      "INFO:tensorflow:Evaluation [3/500]\n",
      "INFO:tensorflow:Evaluation [4/500]\n",
      "INFO:tensorflow:Evaluation [5/500]\n",
      "INFO:tensorflow:Evaluation [6/500]\n",
      "INFO:tensorflow:Evaluation [7/500]\n",
      "INFO:tensorflow:Evaluation [8/500]\n",
      "INFO:tensorflow:Evaluation [9/500]\n",
      "INFO:tensorflow:Evaluation [10/500]\n",
      "INFO:tensorflow:Evaluation [11/500]\n",
      "INFO:tensorflow:Evaluation [12/500]\n",
      "INFO:tensorflow:Evaluation [13/500]\n",
      "INFO:tensorflow:Evaluation [14/500]\n",
      "INFO:tensorflow:Evaluation [15/500]\n",
      "INFO:tensorflow:Evaluation [16/500]\n",
      "INFO:tensorflow:Evaluation [17/500]\n",
      "INFO:tensorflow:Evaluation [18/500]\n",
      "INFO:tensorflow:Evaluation [19/500]\n",
      "INFO:tensorflow:Evaluation [20/500]\n",
      "INFO:tensorflow:Evaluation [21/500]\n",
      "INFO:tensorflow:Evaluation [22/500]\n",
      "INFO:tensorflow:Evaluation [23/500]\n",
      "INFO:tensorflow:Evaluation [24/500]\n",
      "INFO:tensorflow:Evaluation [25/500]\n",
      "INFO:tensorflow:Evaluation [26/500]\n",
      "INFO:tensorflow:Evaluation [27/500]\n",
      "INFO:tensorflow:Evaluation [28/500]\n",
      "INFO:tensorflow:Evaluation [29/500]\n",
      "INFO:tensorflow:Evaluation [30/500]\n",
      "INFO:tensorflow:Evaluation [31/500]\n",
      "INFO:tensorflow:Evaluation [32/500]\n",
      "INFO:tensorflow:Evaluation [33/500]\n",
      "INFO:tensorflow:Evaluation [34/500]\n",
      "INFO:tensorflow:Evaluation [35/500]\n",
      "INFO:tensorflow:Evaluation [36/500]\n",
      "INFO:tensorflow:Evaluation [37/500]\n",
      "INFO:tensorflow:Evaluation [38/500]\n",
      "INFO:tensorflow:Evaluation [39/500]\n",
      "INFO:tensorflow:Evaluation [40/500]\n",
      "INFO:tensorflow:Evaluation [41/500]\n",
      "INFO:tensorflow:Evaluation [42/500]\n",
      "INFO:tensorflow:Evaluation [43/500]\n",
      "INFO:tensorflow:Evaluation [44/500]\n",
      "INFO:tensorflow:Evaluation [45/500]\n",
      "INFO:tensorflow:Evaluation [46/500]\n",
      "INFO:tensorflow:Evaluation [47/500]\n",
      "INFO:tensorflow:Evaluation [48/500]\n",
      "INFO:tensorflow:Evaluation [49/500]\n",
      "INFO:tensorflow:Evaluation [50/500]\n",
      "INFO:tensorflow:Evaluation [51/500]\n",
      "INFO:tensorflow:Evaluation [52/500]\n",
      "INFO:tensorflow:Evaluation [53/500]\n",
      "INFO:tensorflow:Evaluation [54/500]\n",
      "INFO:tensorflow:Evaluation [55/500]\n",
      "INFO:tensorflow:Evaluation [56/500]\n",
      "INFO:tensorflow:Evaluation [57/500]\n",
      "INFO:tensorflow:Evaluation [58/500]\n",
      "INFO:tensorflow:Evaluation [59/500]\n",
      "INFO:tensorflow:Evaluation [60/500]\n",
      "INFO:tensorflow:Evaluation [61/500]\n",
      "INFO:tensorflow:Evaluation [62/500]\n",
      "INFO:tensorflow:Evaluation [63/500]\n",
      "INFO:tensorflow:Evaluation [64/500]\n",
      "INFO:tensorflow:Evaluation [65/500]\n",
      "INFO:tensorflow:Evaluation [66/500]\n",
      "INFO:tensorflow:Evaluation [67/500]\n",
      "INFO:tensorflow:Evaluation [68/500]\n",
      "INFO:tensorflow:Evaluation [69/500]\n",
      "INFO:tensorflow:Evaluation [70/500]\n",
      "INFO:tensorflow:Evaluation [71/500]\n",
      "INFO:tensorflow:Evaluation [72/500]\n",
      "INFO:tensorflow:Evaluation [73/500]\n",
      "INFO:tensorflow:Evaluation [74/500]\n",
      "INFO:tensorflow:Evaluation [75/500]\n",
      "INFO:tensorflow:Evaluation [76/500]\n",
      "INFO:tensorflow:Evaluation [77/500]\n",
      "INFO:tensorflow:Evaluation [78/500]\n",
      "INFO:tensorflow:Evaluation [79/500]\n",
      "INFO:tensorflow:Evaluation [80/500]\n",
      "INFO:tensorflow:Evaluation [81/500]\n",
      "INFO:tensorflow:Evaluation [82/500]\n",
      "INFO:tensorflow:Evaluation [83/500]\n",
      "INFO:tensorflow:Evaluation [84/500]\n",
      "INFO:tensorflow:Evaluation [85/500]\n",
      "INFO:tensorflow:Evaluation [86/500]\n",
      "INFO:tensorflow:Evaluation [87/500]\n",
      "INFO:tensorflow:Evaluation [88/500]\n",
      "INFO:tensorflow:Evaluation [89/500]\n",
      "INFO:tensorflow:Evaluation [90/500]\n",
      "INFO:tensorflow:Evaluation [91/500]\n",
      "INFO:tensorflow:Evaluation [92/500]\n",
      "INFO:tensorflow:Evaluation [93/500]\n",
      "INFO:tensorflow:Evaluation [94/500]\n",
      "INFO:tensorflow:Evaluation [95/500]\n",
      "INFO:tensorflow:Evaluation [96/500]\n",
      "INFO:tensorflow:Evaluation [97/500]\n",
      "INFO:tensorflow:Evaluation [98/500]\n",
      "INFO:tensorflow:Evaluation [99/500]\n",
      "INFO:tensorflow:Evaluation [100/500]\n",
      "INFO:tensorflow:Evaluation [101/500]\n",
      "INFO:tensorflow:Evaluation [102/500]\n",
      "INFO:tensorflow:Evaluation [103/500]\n",
      "INFO:tensorflow:Evaluation [104/500]\n",
      "INFO:tensorflow:Evaluation [105/500]\n",
      "INFO:tensorflow:Evaluation [106/500]\n",
      "INFO:tensorflow:Evaluation [107/500]\n",
      "INFO:tensorflow:Evaluation [108/500]\n",
      "INFO:tensorflow:Evaluation [109/500]\n",
      "INFO:tensorflow:Evaluation [110/500]\n",
      "INFO:tensorflow:Evaluation [111/500]\n",
      "INFO:tensorflow:Evaluation [112/500]\n",
      "INFO:tensorflow:Evaluation [113/500]\n",
      "INFO:tensorflow:Evaluation [114/500]\n",
      "INFO:tensorflow:Evaluation [115/500]\n",
      "INFO:tensorflow:Evaluation [116/500]\n",
      "INFO:tensorflow:Evaluation [117/500]\n",
      "INFO:tensorflow:Evaluation [118/500]\n",
      "INFO:tensorflow:Evaluation [119/500]\n",
      "INFO:tensorflow:Evaluation [120/500]\n",
      "INFO:tensorflow:Evaluation [121/500]\n",
      "INFO:tensorflow:Evaluation [122/500]\n",
      "INFO:tensorflow:Evaluation [123/500]\n",
      "INFO:tensorflow:Evaluation [124/500]\n",
      "INFO:tensorflow:Evaluation [125/500]\n",
      "INFO:tensorflow:Evaluation [126/500]\n",
      "INFO:tensorflow:Evaluation [127/500]\n",
      "INFO:tensorflow:Evaluation [128/500]\n",
      "INFO:tensorflow:Evaluation [129/500]\n",
      "INFO:tensorflow:Evaluation [130/500]\n",
      "INFO:tensorflow:Evaluation [131/500]\n",
      "INFO:tensorflow:Evaluation [132/500]\n",
      "INFO:tensorflow:Evaluation [133/500]\n",
      "INFO:tensorflow:Evaluation [134/500]\n",
      "INFO:tensorflow:Evaluation [135/500]\n",
      "INFO:tensorflow:Evaluation [136/500]\n",
      "INFO:tensorflow:Evaluation [137/500]\n",
      "INFO:tensorflow:Evaluation [138/500]\n",
      "INFO:tensorflow:Evaluation [139/500]\n",
      "INFO:tensorflow:Evaluation [140/500]\n",
      "INFO:tensorflow:Evaluation [141/500]\n",
      "INFO:tensorflow:Evaluation [142/500]\n",
      "INFO:tensorflow:Evaluation [143/500]\n",
      "INFO:tensorflow:Evaluation [144/500]\n",
      "INFO:tensorflow:Evaluation [145/500]\n",
      "INFO:tensorflow:Evaluation [146/500]\n",
      "INFO:tensorflow:Evaluation [147/500]\n",
      "INFO:tensorflow:Evaluation [148/500]\n",
      "INFO:tensorflow:Evaluation [149/500]\n",
      "INFO:tensorflow:Evaluation [150/500]\n",
      "INFO:tensorflow:Evaluation [151/500]\n",
      "INFO:tensorflow:Evaluation [152/500]\n",
      "INFO:tensorflow:Evaluation [153/500]\n",
      "INFO:tensorflow:Evaluation [154/500]\n",
      "INFO:tensorflow:Evaluation [155/500]\n",
      "INFO:tensorflow:Evaluation [156/500]\n",
      "INFO:tensorflow:Evaluation [157/500]\n",
      "INFO:tensorflow:Evaluation [158/500]\n",
      "INFO:tensorflow:Evaluation [159/500]\n",
      "INFO:tensorflow:Evaluation [160/500]\n",
      "INFO:tensorflow:Evaluation [161/500]\n",
      "INFO:tensorflow:Evaluation [162/500]\n",
      "INFO:tensorflow:Evaluation [163/500]\n",
      "INFO:tensorflow:Evaluation [164/500]\n",
      "INFO:tensorflow:Evaluation [165/500]\n",
      "INFO:tensorflow:Evaluation [166/500]\n",
      "INFO:tensorflow:Evaluation [167/500]\n",
      "INFO:tensorflow:Evaluation [168/500]\n",
      "INFO:tensorflow:Evaluation [169/500]\n",
      "INFO:tensorflow:Evaluation [170/500]\n",
      "INFO:tensorflow:Evaluation [171/500]\n",
      "INFO:tensorflow:Evaluation [172/500]\n",
      "INFO:tensorflow:Evaluation [173/500]\n",
      "INFO:tensorflow:Evaluation [174/500]\n",
      "INFO:tensorflow:Evaluation [175/500]\n",
      "INFO:tensorflow:Evaluation [176/500]\n",
      "INFO:tensorflow:Evaluation [177/500]\n",
      "INFO:tensorflow:Evaluation [178/500]\n",
      "INFO:tensorflow:Evaluation [179/500]\n",
      "INFO:tensorflow:Evaluation [180/500]\n",
      "INFO:tensorflow:Evaluation [181/500]\n",
      "INFO:tensorflow:Evaluation [182/500]\n",
      "INFO:tensorflow:Evaluation [183/500]\n",
      "INFO:tensorflow:Evaluation [184/500]\n",
      "INFO:tensorflow:Evaluation [185/500]\n",
      "INFO:tensorflow:Evaluation [186/500]\n",
      "INFO:tensorflow:Evaluation [187/500]\n",
      "INFO:tensorflow:Evaluation [188/500]\n",
      "INFO:tensorflow:Evaluation [189/500]\n",
      "INFO:tensorflow:Evaluation [190/500]\n",
      "INFO:tensorflow:Evaluation [191/500]\n",
      "INFO:tensorflow:Evaluation [192/500]\n",
      "INFO:tensorflow:Evaluation [193/500]\n",
      "INFO:tensorflow:Evaluation [194/500]\n",
      "INFO:tensorflow:Evaluation [195/500]\n",
      "INFO:tensorflow:Evaluation [196/500]\n",
      "INFO:tensorflow:Evaluation [197/500]\n",
      "INFO:tensorflow:Evaluation [198/500]\n",
      "INFO:tensorflow:Evaluation [199/500]\n",
      "INFO:tensorflow:Evaluation [200/500]\n",
      "INFO:tensorflow:Evaluation [201/500]\n",
      "INFO:tensorflow:Evaluation [202/500]\n",
      "INFO:tensorflow:Evaluation [203/500]\n",
      "INFO:tensorflow:Evaluation [204/500]\n",
      "INFO:tensorflow:Evaluation [205/500]\n",
      "INFO:tensorflow:Evaluation [206/500]\n",
      "INFO:tensorflow:Evaluation [207/500]\n",
      "INFO:tensorflow:Evaluation [208/500]\n",
      "INFO:tensorflow:Evaluation [209/500]\n",
      "INFO:tensorflow:Evaluation [210/500]\n",
      "INFO:tensorflow:Evaluation [211/500]\n",
      "INFO:tensorflow:Evaluation [212/500]\n",
      "INFO:tensorflow:Evaluation [213/500]\n",
      "INFO:tensorflow:Evaluation [214/500]\n",
      "INFO:tensorflow:Evaluation [215/500]\n",
      "INFO:tensorflow:Evaluation [216/500]\n",
      "INFO:tensorflow:Evaluation [217/500]\n",
      "INFO:tensorflow:Evaluation [218/500]\n",
      "INFO:tensorflow:Evaluation [219/500]\n",
      "INFO:tensorflow:Evaluation [220/500]\n",
      "INFO:tensorflow:Evaluation [221/500]\n",
      "INFO:tensorflow:Evaluation [222/500]\n",
      "INFO:tensorflow:Evaluation [223/500]\n",
      "INFO:tensorflow:Evaluation [224/500]\n",
      "INFO:tensorflow:Evaluation [225/500]\n",
      "INFO:tensorflow:Evaluation [226/500]\n",
      "INFO:tensorflow:Evaluation [227/500]\n",
      "INFO:tensorflow:Evaluation [228/500]\n",
      "INFO:tensorflow:Evaluation [229/500]\n",
      "INFO:tensorflow:Evaluation [230/500]\n",
      "INFO:tensorflow:Evaluation [231/500]\n",
      "INFO:tensorflow:Evaluation [232/500]\n",
      "INFO:tensorflow:Evaluation [233/500]\n",
      "INFO:tensorflow:Evaluation [234/500]\n",
      "INFO:tensorflow:Evaluation [235/500]\n",
      "INFO:tensorflow:Evaluation [236/500]\n",
      "INFO:tensorflow:Evaluation [237/500]\n",
      "INFO:tensorflow:Evaluation [238/500]\n",
      "INFO:tensorflow:Evaluation [239/500]\n",
      "INFO:tensorflow:Evaluation [240/500]\n",
      "INFO:tensorflow:Evaluation [241/500]\n",
      "INFO:tensorflow:Evaluation [242/500]\n",
      "INFO:tensorflow:Evaluation [243/500]\n",
      "INFO:tensorflow:Evaluation [244/500]\n",
      "INFO:tensorflow:Evaluation [245/500]\n",
      "INFO:tensorflow:Evaluation [246/500]\n",
      "INFO:tensorflow:Evaluation [247/500]\n",
      "INFO:tensorflow:Evaluation [248/500]\n",
      "INFO:tensorflow:Evaluation [249/500]\n",
      "INFO:tensorflow:Evaluation [250/500]\n",
      "INFO:tensorflow:Evaluation [251/500]\n",
      "INFO:tensorflow:Evaluation [252/500]\n",
      "INFO:tensorflow:Evaluation [253/500]\n",
      "INFO:tensorflow:Evaluation [254/500]\n",
      "INFO:tensorflow:Evaluation [255/500]\n",
      "INFO:tensorflow:Evaluation [256/500]\n",
      "INFO:tensorflow:Evaluation [257/500]\n",
      "INFO:tensorflow:Evaluation [258/500]\n",
      "INFO:tensorflow:Evaluation [259/500]\n",
      "INFO:tensorflow:Evaluation [260/500]\n",
      "INFO:tensorflow:Evaluation [261/500]\n",
      "INFO:tensorflow:Evaluation [262/500]\n",
      "INFO:tensorflow:Evaluation [263/500]\n",
      "INFO:tensorflow:Evaluation [264/500]\n",
      "INFO:tensorflow:Evaluation [265/500]\n",
      "INFO:tensorflow:Evaluation [266/500]\n",
      "INFO:tensorflow:Evaluation [267/500]\n",
      "INFO:tensorflow:Evaluation [268/500]\n",
      "INFO:tensorflow:Evaluation [269/500]\n",
      "INFO:tensorflow:Evaluation [270/500]\n",
      "INFO:tensorflow:Evaluation [271/500]\n",
      "INFO:tensorflow:Evaluation [272/500]\n",
      "INFO:tensorflow:Evaluation [273/500]\n",
      "INFO:tensorflow:Evaluation [274/500]\n",
      "INFO:tensorflow:Evaluation [275/500]\n",
      "INFO:tensorflow:Evaluation [276/500]\n",
      "INFO:tensorflow:Evaluation [277/500]\n",
      "INFO:tensorflow:Evaluation [278/500]\n",
      "INFO:tensorflow:Evaluation [279/500]\n",
      "INFO:tensorflow:Evaluation [280/500]\n",
      "INFO:tensorflow:Evaluation [281/500]\n",
      "INFO:tensorflow:Evaluation [282/500]\n",
      "INFO:tensorflow:Evaluation [283/500]\n",
      "INFO:tensorflow:Evaluation [284/500]\n",
      "INFO:tensorflow:Evaluation [285/500]\n",
      "INFO:tensorflow:Evaluation [286/500]\n",
      "INFO:tensorflow:Evaluation [287/500]\n",
      "INFO:tensorflow:Evaluation [288/500]\n",
      "INFO:tensorflow:Evaluation [289/500]\n",
      "INFO:tensorflow:Evaluation [290/500]\n",
      "INFO:tensorflow:Evaluation [291/500]\n",
      "INFO:tensorflow:Evaluation [292/500]\n",
      "INFO:tensorflow:Evaluation [293/500]\n",
      "INFO:tensorflow:Evaluation [294/500]\n",
      "INFO:tensorflow:Evaluation [295/500]\n",
      "INFO:tensorflow:Evaluation [296/500]\n",
      "INFO:tensorflow:Evaluation [297/500]\n",
      "INFO:tensorflow:Evaluation [298/500]\n",
      "INFO:tensorflow:Evaluation [299/500]\n",
      "INFO:tensorflow:Evaluation [300/500]\n",
      "INFO:tensorflow:Evaluation [301/500]\n",
      "INFO:tensorflow:Evaluation [302/500]\n",
      "INFO:tensorflow:Evaluation [303/500]\n",
      "INFO:tensorflow:Evaluation [304/500]\n",
      "INFO:tensorflow:Evaluation [305/500]\n",
      "INFO:tensorflow:Evaluation [306/500]\n",
      "INFO:tensorflow:Evaluation [307/500]\n",
      "INFO:tensorflow:Evaluation [308/500]\n",
      "INFO:tensorflow:Evaluation [309/500]\n",
      "INFO:tensorflow:Evaluation [310/500]\n",
      "INFO:tensorflow:Evaluation [311/500]\n",
      "INFO:tensorflow:Evaluation [312/500]\n",
      "INFO:tensorflow:Evaluation [313/500]\n",
      "INFO:tensorflow:Evaluation [314/500]\n",
      "INFO:tensorflow:Evaluation [315/500]\n",
      "INFO:tensorflow:Evaluation [316/500]\n",
      "INFO:tensorflow:Evaluation [317/500]\n",
      "INFO:tensorflow:Evaluation [318/500]\n",
      "INFO:tensorflow:Evaluation [319/500]\n",
      "INFO:tensorflow:Evaluation [320/500]\n",
      "INFO:tensorflow:Evaluation [321/500]\n",
      "INFO:tensorflow:Evaluation [322/500]\n",
      "INFO:tensorflow:Evaluation [323/500]\n",
      "INFO:tensorflow:Evaluation [324/500]\n",
      "INFO:tensorflow:Evaluation [325/500]\n",
      "INFO:tensorflow:Evaluation [326/500]\n",
      "INFO:tensorflow:Evaluation [327/500]\n",
      "INFO:tensorflow:Evaluation [328/500]\n",
      "INFO:tensorflow:Evaluation [329/500]\n",
      "INFO:tensorflow:Evaluation [330/500]\n",
      "INFO:tensorflow:Evaluation [331/500]\n",
      "INFO:tensorflow:Evaluation [332/500]\n",
      "INFO:tensorflow:Evaluation [333/500]\n",
      "INFO:tensorflow:Evaluation [334/500]\n",
      "INFO:tensorflow:Evaluation [335/500]\n",
      "INFO:tensorflow:Evaluation [336/500]\n",
      "INFO:tensorflow:Evaluation [337/500]\n",
      "INFO:tensorflow:Evaluation [338/500]\n",
      "INFO:tensorflow:Evaluation [339/500]\n",
      "INFO:tensorflow:Evaluation [340/500]\n",
      "INFO:tensorflow:Evaluation [341/500]\n",
      "INFO:tensorflow:Evaluation [342/500]\n",
      "INFO:tensorflow:Evaluation [343/500]\n",
      "INFO:tensorflow:Evaluation [344/500]\n",
      "INFO:tensorflow:Evaluation [345/500]\n",
      "INFO:tensorflow:Evaluation [346/500]\n",
      "INFO:tensorflow:Evaluation [347/500]\n",
      "INFO:tensorflow:Evaluation [348/500]\n",
      "INFO:tensorflow:Evaluation [349/500]\n",
      "INFO:tensorflow:Evaluation [350/500]\n",
      "INFO:tensorflow:Evaluation [351/500]\n",
      "INFO:tensorflow:Evaluation [352/500]\n",
      "INFO:tensorflow:Evaluation [353/500]\n",
      "INFO:tensorflow:Evaluation [354/500]\n",
      "INFO:tensorflow:Evaluation [355/500]\n",
      "INFO:tensorflow:Evaluation [356/500]\n",
      "INFO:tensorflow:Evaluation [357/500]\n",
      "INFO:tensorflow:Evaluation [358/500]\n",
      "INFO:tensorflow:Evaluation [359/500]\n",
      "INFO:tensorflow:Evaluation [360/500]\n",
      "INFO:tensorflow:Evaluation [361/500]\n",
      "INFO:tensorflow:Evaluation [362/500]\n",
      "INFO:tensorflow:Evaluation [363/500]\n",
      "INFO:tensorflow:Evaluation [364/500]\n",
      "INFO:tensorflow:Evaluation [365/500]\n",
      "INFO:tensorflow:Evaluation [366/500]\n",
      "INFO:tensorflow:Evaluation [367/500]\n",
      "INFO:tensorflow:Evaluation [368/500]\n",
      "INFO:tensorflow:Evaluation [369/500]\n",
      "INFO:tensorflow:Evaluation [370/500]\n",
      "INFO:tensorflow:Evaluation [371/500]\n",
      "INFO:tensorflow:Evaluation [372/500]\n",
      "INFO:tensorflow:Evaluation [373/500]\n",
      "INFO:tensorflow:Evaluation [374/500]\n",
      "INFO:tensorflow:Evaluation [375/500]\n",
      "INFO:tensorflow:Evaluation [376/500]\n",
      "INFO:tensorflow:Evaluation [377/500]\n",
      "INFO:tensorflow:Evaluation [378/500]\n",
      "INFO:tensorflow:Evaluation [379/500]\n",
      "INFO:tensorflow:Evaluation [380/500]\n",
      "INFO:tensorflow:Evaluation [381/500]\n",
      "INFO:tensorflow:Evaluation [382/500]\n",
      "INFO:tensorflow:Evaluation [383/500]\n",
      "INFO:tensorflow:Evaluation [384/500]\n",
      "INFO:tensorflow:Evaluation [385/500]\n",
      "INFO:tensorflow:Evaluation [386/500]\n",
      "INFO:tensorflow:Evaluation [387/500]\n",
      "INFO:tensorflow:Evaluation [388/500]\n",
      "INFO:tensorflow:Evaluation [389/500]\n",
      "INFO:tensorflow:Evaluation [390/500]\n",
      "INFO:tensorflow:Evaluation [391/500]\n",
      "INFO:tensorflow:Evaluation [392/500]\n",
      "INFO:tensorflow:Evaluation [393/500]\n",
      "INFO:tensorflow:Evaluation [394/500]\n",
      "INFO:tensorflow:Evaluation [395/500]\n",
      "INFO:tensorflow:Evaluation [396/500]\n",
      "INFO:tensorflow:Evaluation [397/500]\n",
      "INFO:tensorflow:Evaluation [398/500]\n",
      "INFO:tensorflow:Evaluation [399/500]\n",
      "INFO:tensorflow:Evaluation [400/500]\n",
      "INFO:tensorflow:Evaluation [401/500]\n",
      "INFO:tensorflow:Evaluation [402/500]\n",
      "INFO:tensorflow:Evaluation [403/500]\n",
      "INFO:tensorflow:Evaluation [404/500]\n",
      "INFO:tensorflow:Evaluation [405/500]\n",
      "INFO:tensorflow:Evaluation [406/500]\n",
      "INFO:tensorflow:Evaluation [407/500]\n",
      "INFO:tensorflow:Evaluation [408/500]\n",
      "INFO:tensorflow:Evaluation [409/500]\n",
      "INFO:tensorflow:Evaluation [410/500]\n",
      "INFO:tensorflow:Evaluation [411/500]\n",
      "INFO:tensorflow:Evaluation [412/500]\n",
      "INFO:tensorflow:Evaluation [413/500]\n",
      "INFO:tensorflow:Evaluation [414/500]\n",
      "INFO:tensorflow:Evaluation [415/500]\n",
      "INFO:tensorflow:Evaluation [416/500]\n",
      "INFO:tensorflow:Evaluation [417/500]\n",
      "INFO:tensorflow:Evaluation [418/500]\n",
      "INFO:tensorflow:Evaluation [419/500]\n",
      "INFO:tensorflow:Evaluation [420/500]\n",
      "INFO:tensorflow:Evaluation [421/500]\n",
      "INFO:tensorflow:Evaluation [422/500]\n",
      "INFO:tensorflow:Evaluation [423/500]\n",
      "INFO:tensorflow:Evaluation [424/500]\n",
      "INFO:tensorflow:Evaluation [425/500]\n",
      "INFO:tensorflow:Evaluation [426/500]\n",
      "INFO:tensorflow:Evaluation [427/500]\n",
      "INFO:tensorflow:Evaluation [428/500]\n",
      "INFO:tensorflow:Evaluation [429/500]\n",
      "INFO:tensorflow:Evaluation [430/500]\n",
      "INFO:tensorflow:Evaluation [431/500]\n",
      "INFO:tensorflow:Evaluation [432/500]\n",
      "INFO:tensorflow:Evaluation [433/500]\n",
      "INFO:tensorflow:Evaluation [434/500]\n",
      "INFO:tensorflow:Evaluation [435/500]\n",
      "INFO:tensorflow:Evaluation [436/500]\n",
      "INFO:tensorflow:Evaluation [437/500]\n",
      "INFO:tensorflow:Evaluation [438/500]\n",
      "INFO:tensorflow:Evaluation [439/500]\n",
      "INFO:tensorflow:Evaluation [440/500]\n",
      "INFO:tensorflow:Evaluation [441/500]\n",
      "INFO:tensorflow:Evaluation [442/500]\n",
      "INFO:tensorflow:Evaluation [443/500]\n",
      "INFO:tensorflow:Evaluation [444/500]\n",
      "INFO:tensorflow:Evaluation [445/500]\n",
      "INFO:tensorflow:Evaluation [446/500]\n",
      "INFO:tensorflow:Evaluation [447/500]\n",
      "INFO:tensorflow:Evaluation [448/500]\n",
      "INFO:tensorflow:Evaluation [449/500]\n",
      "INFO:tensorflow:Evaluation [450/500]\n",
      "INFO:tensorflow:Evaluation [451/500]\n",
      "INFO:tensorflow:Evaluation [452/500]\n",
      "INFO:tensorflow:Evaluation [453/500]\n",
      "INFO:tensorflow:Evaluation [454/500]\n",
      "INFO:tensorflow:Evaluation [455/500]\n",
      "INFO:tensorflow:Evaluation [456/500]\n",
      "INFO:tensorflow:Evaluation [457/500]\n",
      "INFO:tensorflow:Evaluation [458/500]\n",
      "INFO:tensorflow:Evaluation [459/500]\n",
      "INFO:tensorflow:Evaluation [460/500]\n",
      "INFO:tensorflow:Evaluation [461/500]\n",
      "INFO:tensorflow:Evaluation [462/500]\n",
      "INFO:tensorflow:Evaluation [463/500]\n",
      "INFO:tensorflow:Evaluation [464/500]\n",
      "INFO:tensorflow:Evaluation [465/500]\n",
      "INFO:tensorflow:Evaluation [466/500]\n",
      "INFO:tensorflow:Evaluation [467/500]\n",
      "INFO:tensorflow:Evaluation [468/500]\n",
      "INFO:tensorflow:Evaluation [469/500]\n",
      "INFO:tensorflow:Evaluation [470/500]\n",
      "INFO:tensorflow:Evaluation [471/500]\n",
      "INFO:tensorflow:Evaluation [472/500]\n",
      "INFO:tensorflow:Evaluation [473/500]\n",
      "INFO:tensorflow:Evaluation [474/500]\n",
      "INFO:tensorflow:Evaluation [475/500]\n",
      "INFO:tensorflow:Evaluation [476/500]\n",
      "INFO:tensorflow:Evaluation [477/500]\n",
      "INFO:tensorflow:Evaluation [478/500]\n",
      "INFO:tensorflow:Evaluation [479/500]\n",
      "INFO:tensorflow:Evaluation [480/500]\n",
      "INFO:tensorflow:Evaluation [481/500]\n",
      "INFO:tensorflow:Evaluation [482/500]\n",
      "INFO:tensorflow:Evaluation [483/500]\n",
      "INFO:tensorflow:Evaluation [484/500]\n",
      "INFO:tensorflow:Evaluation [485/500]\n",
      "INFO:tensorflow:Evaluation [486/500]\n",
      "INFO:tensorflow:Evaluation [487/500]\n",
      "INFO:tensorflow:Evaluation [488/500]\n",
      "INFO:tensorflow:Evaluation [489/500]\n",
      "INFO:tensorflow:Evaluation [490/500]\n",
      "INFO:tensorflow:Evaluation [491/500]\n",
      "INFO:tensorflow:Evaluation [492/500]\n",
      "INFO:tensorflow:Evaluation [493/500]\n",
      "INFO:tensorflow:Evaluation [494/500]\n",
      "INFO:tensorflow:Evaluation [495/500]\n",
      "INFO:tensorflow:Evaluation [496/500]\n",
      "INFO:tensorflow:Evaluation [497/500]\n",
      "INFO:tensorflow:Evaluation [498/500]\n",
      "INFO:tensorflow:Evaluation [499/500]\n",
      "INFO:tensorflow:Evaluation [500/500]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-02-04:09:36\n",
      "INFO:tensorflow:Saving dict for global step 2002: accuracy = 0.7905, accuracy/baseline_label_mean = 0.209, accuracy/threshold_0.500000_mean = 0.7905, auc = 0.71043, auc_precision_recall = 0.375325, global_step = 2002, labels/actual_label_mean = 0.209, labels/prediction_mean = 0.211079, loss = 0.469096, precision/positive_threshold_0.500000_mean = 0.495146, recall/positive_threshold_0.500000_mean = 0.12201\n",
      "evaluate done\n",
      "Accuracy: 0.7905\n",
      "{'loss': 0.46909648, 'accuracy': 0.79049999, 'labels/prediction_mean': 0.21107854, 'labels/actual_label_mean': 0.20900001, 'accuracy/baseline_label_mean': 0.20900001, 'auc': 0.71043026, 'auc_precision_recall': 0.37532485, 'accuracy/threshold_0.500000_mean': 0.79049999, 'precision/positive_threshold_0.500000_mean': 0.49514562, 'recall/positive_threshold_0.500000_mean': 0.12200957, 'global_step': 2002}\n",
      "Wall time: 20 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "eval_sample_size = 200000 # this can be found with a 'wc -l eval.csv'\n",
    "eval_steps = eval_sample_size/BATCH_SIZE # 2000/40 = 50\n",
    "\n",
    "results = m.evaluate(input_fn=generate_input_fn(eval_file), \n",
    "                     steps=eval_steps)\n",
    "print('evaluate done')\n",
    "\n",
    "print('Accuracy: %s' % results['accuracy'])\n",
    "print(results)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Export a model optimized for inference\n",
    "We can upload our trained model to the Cloud Machine Learning Engine's Prediction Service, which will take care of serving our model and scaling it. The code below exports our trained model to a `saved_model.pb` file and a `variables` folder where the trained weights are stored. \n",
    "\n",
    "The `export_savedmodel()` function expects a `serving_input_fn()`, which returns the mapping from the data that the Prediction Service passes in to the data that should be fed into the trained TensorFlow prediction graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.contrib.learn.python.learn.utils import input_fn_utils\n",
    "\n",
    "def column_to_dtype(column):\n",
    "    if column in CATEGORICAL_COLUMNS:\n",
    "        return tf.string\n",
    "    else:\n",
    "        return tf.float32\n",
    "\n",
    "def serving_input_fn():\n",
    "    feature_placeholders = {\n",
    "        column: tf.placeholder(column_to_dtype(column), [None])\n",
    "        for column in FEATURE_COLUMNS\n",
    "    }\n",
    "    # DNNCombinedLinearClassifier expects rank 2 Tensors, but inputs should be\n",
    "    # rank 1, so that we can provide scalars to the server\n",
    "    features = {\n",
    "        key: tf.expand_dims(tensor, -1)\n",
    "        for key, tensor in feature_placeholders.items()\n",
    "    }\n",
    "    \n",
    "    return input_fn_utils.InputFnOps(\n",
    "        features, # input into graph\n",
    "        None,\n",
    "        feature_placeholders # tensor input converted from request \n",
    "    )\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "export_savedmodel() got an unexpected keyword argument 'input_fn'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-15-46ba0866336f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m export_folder = m.export_savedmodel(\n\u001b[1;32m      3\u001b[0m     \u001b[0mexport_dir_base\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmodel_dir\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m'/export'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m     \u001b[0minput_fn\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mserving_input_fn\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m )\n\u001b[1;32m      6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: export_savedmodel() got an unexpected keyword argument 'input_fn'"
     ]
    }
   ],
   "source": [
    "# Manually export\n",
    "export_folder = m.export_savedmodel(\n",
    "    export_dir_base = model_dir + '/export',\n",
    "    input_fn=serving_input_fn\n",
    ")\n",
    "\n",
    "print('model exported successfully to {}'.format(export_folder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Experiments to manage the training workflow\n",
    "TensorFlow also offers an \"Experiments\" framework to help manage your training for you. Using it will take away some of the boilerplate and automate certain actions, but it's important to understand what is happening (what we did above) before using something like this.\n",
    "\n",
    "Notice that we don't need to create additional wrapper functions here, but rather, we can directly use the various functions we already defined previously. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from tensorflow.contrib.learn.python.learn import learn_runner\n",
    "from tensorflow.contrib.learn.python.learn.utils import saved_model_export_utils\n",
    "\n",
    "\n",
    "# get_model(model_type = 'WIDE_AND_DEEP', model_dir=model_dir)\n",
    "\n",
    "# output_dir is an arg passed in by the learn_runner.run() call.\n",
    "def experiment_fn(output_dir):\n",
    "    \n",
    "    print(output_dir)\n",
    "    \n",
    "    train_input_fn = generate_input_fn(train_file, BATCH_SIZE)\n",
    "    eval_input_fn = generate_input_fn(eval_file)\n",
    "    my_model = get_model(model_type=MODEL_TYPE, \n",
    "                  model_dir=output_dir)\n",
    "\n",
    "    experiment = tf.contrib.learn.Experiment(\n",
    "        my_model,\n",
    "        train_input_fn=train_input_fn,\n",
    "        eval_input_fn=eval_input_fn,\n",
    "        train_steps=1000\n",
    "        ,\n",
    "        export_strategies=[saved_model_export_utils.make_export_strategy(\n",
    "            serving_input_fn,\n",
    "            default_output_alternative_key=None,\n",
    "            exports_to_keep=1\n",
    "        )]\n",
    "    )\n",
    "    return experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-17-c873d63bd43e>, line 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-17-c873d63bd43e>\"\u001b[0;36m, line \u001b[0;32m2\u001b[0m\n\u001b[0;31m    %%time\u001b[0m\n\u001b[0m    ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# manually train and eval\n",
    "%%time\n",
    "\n",
    "exp = experiment_fn(model_dir)\n",
    "\n",
    "exp.train_and_evaluate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "models/model_WIDE_AND_DEEP_1556720127\n",
      "Model directory = models/model_WIDE_AND_DEEP_1556720127\n",
      "WARNING:tensorflow:From <ipython-input-8-6aeab337b09f>:40: calling DNNLinearCombinedClassifier.__init__ (from tensorflow.contrib.learn.python.learn.estimators.dnn_linear_combined) with fix_global_step_increment_bug=False is deprecated and will be removed after 2017-04-15.\n",
      "Instructions for updating:\n",
      "Please set fix_global_step_increment_bug=True and update training steps in your pipeline. See pydoc for details.\n",
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001D1C6054278>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_session_config': None, '_save_checkpoints_steps': 100, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'models/model_WIDE_AND_DEEP_1556720127'}\n",
      "estimator built\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\monitors.py:268: BaseMonitor.__init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.\n",
      "Instructions for updating:\n",
      "Monitors are deprecated. Please use tf.train.SessionRunHook.\n",
      "features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 2 into models/model_WIDE_AND_DEEP_1556720127\\model.ckpt.\n",
      "INFO:tensorflow:loss = 38.3895, step = 2\n",
      "features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-01-14:16:11\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556720127\\model.ckpt-2\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Evaluation [39/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [41/100]\n",
      "INFO:tensorflow:Evaluation [42/100]\n",
      "INFO:tensorflow:Evaluation [43/100]\n",
      "INFO:tensorflow:Evaluation [44/100]\n",
      "INFO:tensorflow:Evaluation [45/100]\n",
      "INFO:tensorflow:Evaluation [46/100]\n",
      "INFO:tensorflow:Evaluation [47/100]\n",
      "INFO:tensorflow:Evaluation [48/100]\n",
      "INFO:tensorflow:Evaluation [49/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [51/100]\n",
      "INFO:tensorflow:Evaluation [52/100]\n",
      "INFO:tensorflow:Evaluation [53/100]\n",
      "INFO:tensorflow:Evaluation [54/100]\n",
      "INFO:tensorflow:Evaluation [55/100]\n",
      "INFO:tensorflow:Evaluation [56/100]\n",
      "INFO:tensorflow:Evaluation [57/100]\n",
      "INFO:tensorflow:Evaluation [58/100]\n",
      "INFO:tensorflow:Evaluation [59/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [61/100]\n",
      "INFO:tensorflow:Evaluation [62/100]\n",
      "INFO:tensorflow:Evaluation [63/100]\n",
      "INFO:tensorflow:Evaluation [64/100]\n",
      "INFO:tensorflow:Evaluation [65/100]\n",
      "INFO:tensorflow:Evaluation [66/100]\n",
      "INFO:tensorflow:Evaluation [67/100]\n",
      "INFO:tensorflow:Evaluation [68/100]\n",
      "INFO:tensorflow:Evaluation [69/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [71/100]\n",
      "INFO:tensorflow:Evaluation [72/100]\n",
      "INFO:tensorflow:Evaluation [73/100]\n",
      "INFO:tensorflow:Evaluation [74/100]\n",
      "INFO:tensorflow:Evaluation [75/100]\n",
      "INFO:tensorflow:Evaluation [76/100]\n",
      "INFO:tensorflow:Evaluation [77/100]\n",
      "INFO:tensorflow:Evaluation [78/100]\n",
      "INFO:tensorflow:Evaluation [79/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [81/100]\n",
      "INFO:tensorflow:Evaluation [82/100]\n",
      "INFO:tensorflow:Evaluation [83/100]\n",
      "INFO:tensorflow:Evaluation [84/100]\n",
      "INFO:tensorflow:Evaluation [85/100]\n",
      "INFO:tensorflow:Evaluation [86/100]\n",
      "INFO:tensorflow:Evaluation [87/100]\n",
      "INFO:tensorflow:Evaluation [88/100]\n",
      "INFO:tensorflow:Evaluation [89/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [91/100]\n",
      "INFO:tensorflow:Evaluation [92/100]\n",
      "INFO:tensorflow:Evaluation [93/100]\n",
      "INFO:tensorflow:Evaluation [94/100]\n",
      "INFO:tensorflow:Evaluation [95/100]\n",
      "INFO:tensorflow:Evaluation [96/100]\n",
      "INFO:tensorflow:Evaluation [97/100]\n",
      "INFO:tensorflow:Evaluation [98/100]\n",
      "INFO:tensorflow:Evaluation [99/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-01-14:16:15\n",
      "INFO:tensorflow:Saving dict for global step 2: accuracy = 0.209, accuracy/baseline_label_mean = 0.209, accuracy/threshold_0.500000_mean = 0.209, auc = 0.503881, auc_precision_recall = 0.595699, global_step = 2, labels/actual_label_mean = 0.209, labels/prediction_mean = 0.995837, loss = 7041.76, precision/positive_threshold_0.500000_mean = 0.209, recall/positive_threshold_0.500000_mean = 1.0\n",
      "INFO:tensorflow:Validation (step 1): loss = 7041.76, accuracy = 0.209, labels/prediction_mean = 0.995837, labels/actual_label_mean = 0.209, accuracy/baseline_label_mean = 0.209, auc = 0.503881, auc_precision_recall = 0.595699, accuracy/threshold_0.500000_mean = 0.209, precision/positive_threshold_0.500000_mean = 0.209, recall/positive_threshold_0.500000_mean = 1.0, global_step = 2\n",
      "INFO:tensorflow:Saving checkpoints for 102 into models/model_WIDE_AND_DEEP_1556720127\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 4.64246\n",
      "features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-01-14:16:33\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556720127\\model.ckpt-102\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Evaluation [39/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [41/100]\n",
      "INFO:tensorflow:Evaluation [42/100]\n",
      "INFO:tensorflow:Evaluation [43/100]\n",
      "INFO:tensorflow:Evaluation [44/100]\n",
      "INFO:tensorflow:Evaluation [45/100]\n",
      "INFO:tensorflow:Evaluation [46/100]\n",
      "INFO:tensorflow:Evaluation [47/100]\n",
      "INFO:tensorflow:Evaluation [48/100]\n",
      "INFO:tensorflow:Evaluation [49/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [51/100]\n",
      "INFO:tensorflow:Evaluation [52/100]\n",
      "INFO:tensorflow:Evaluation [53/100]\n",
      "INFO:tensorflow:Evaluation [54/100]\n",
      "INFO:tensorflow:Evaluation [55/100]\n",
      "INFO:tensorflow:Evaluation [56/100]\n",
      "INFO:tensorflow:Evaluation [57/100]\n",
      "INFO:tensorflow:Evaluation [58/100]\n",
      "INFO:tensorflow:Evaluation [59/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [61/100]\n",
      "INFO:tensorflow:Evaluation [62/100]\n",
      "INFO:tensorflow:Evaluation [63/100]\n",
      "INFO:tensorflow:Evaluation [64/100]\n",
      "INFO:tensorflow:Evaluation [65/100]\n",
      "INFO:tensorflow:Evaluation [66/100]\n",
      "INFO:tensorflow:Evaluation [67/100]\n",
      "INFO:tensorflow:Evaluation [68/100]\n",
      "INFO:tensorflow:Evaluation [69/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [71/100]\n",
      "INFO:tensorflow:Evaluation [72/100]\n",
      "INFO:tensorflow:Evaluation [73/100]\n",
      "INFO:tensorflow:Evaluation [74/100]\n",
      "INFO:tensorflow:Evaluation [75/100]\n",
      "INFO:tensorflow:Evaluation [76/100]\n",
      "INFO:tensorflow:Evaluation [77/100]\n",
      "INFO:tensorflow:Evaluation [78/100]\n",
      "INFO:tensorflow:Evaluation [79/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [81/100]\n",
      "INFO:tensorflow:Evaluation [82/100]\n",
      "INFO:tensorflow:Evaluation [83/100]\n",
      "INFO:tensorflow:Evaluation [84/100]\n",
      "INFO:tensorflow:Evaluation [85/100]\n",
      "INFO:tensorflow:Evaluation [86/100]\n",
      "INFO:tensorflow:Evaluation [87/100]\n",
      "INFO:tensorflow:Evaluation [88/100]\n",
      "INFO:tensorflow:Evaluation [89/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [91/100]\n",
      "INFO:tensorflow:Evaluation [92/100]\n",
      "INFO:tensorflow:Evaluation [93/100]\n",
      "INFO:tensorflow:Evaluation [94/100]\n",
      "INFO:tensorflow:Evaluation [95/100]\n",
      "INFO:tensorflow:Evaluation [96/100]\n",
      "INFO:tensorflow:Evaluation [97/100]\n",
      "INFO:tensorflow:Evaluation [98/100]\n",
      "INFO:tensorflow:Evaluation [99/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-01-14:16:37\n",
      "INFO:tensorflow:Saving dict for global step 102: accuracy = 0.7715, accuracy/baseline_label_mean = 0.209, accuracy/threshold_0.500000_mean = 0.7715, auc = 0.628811, auc_precision_recall = 0.319358, global_step = 102, labels/actual_label_mean = 0.209, labels/prediction_mean = 0.146524, loss = 2.95424, precision/positive_threshold_0.500000_mean = 0.398964, recall/positive_threshold_0.500000_mean = 0.184211\n",
      "INFO:tensorflow:Validation (step 101): loss = 2.95424, accuracy = 0.7715, labels/prediction_mean = 0.146524, labels/actual_label_mean = 0.209, accuracy/baseline_label_mean = 0.209, auc = 0.628811, auc_precision_recall = 0.319358, accuracy/threshold_0.500000_mean = 0.7715, precision/positive_threshold_0.500000_mean = 0.398964, recall/positive_threshold_0.500000_mean = 0.184211, global_step = 102\n",
      "INFO:tensorflow:Saving checkpoints for 202 into models/model_WIDE_AND_DEEP_1556720127\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 6.70276\n",
      "INFO:tensorflow:loss = 1.30062, step = 202 (36.458 sec)\n",
      "features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-01-14:16:48\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556720127\\model.ckpt-202\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Evaluation [39/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [41/100]\n",
      "INFO:tensorflow:Evaluation [42/100]\n",
      "INFO:tensorflow:Evaluation [43/100]\n",
      "INFO:tensorflow:Evaluation [44/100]\n",
      "INFO:tensorflow:Evaluation [45/100]\n",
      "INFO:tensorflow:Evaluation [46/100]\n",
      "INFO:tensorflow:Evaluation [47/100]\n",
      "INFO:tensorflow:Evaluation [48/100]\n",
      "INFO:tensorflow:Evaluation [49/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [51/100]\n",
      "INFO:tensorflow:Evaluation [52/100]\n",
      "INFO:tensorflow:Evaluation [53/100]\n",
      "INFO:tensorflow:Evaluation [54/100]\n",
      "INFO:tensorflow:Evaluation [55/100]\n",
      "INFO:tensorflow:Evaluation [56/100]\n",
      "INFO:tensorflow:Evaluation [57/100]\n",
      "INFO:tensorflow:Evaluation [58/100]\n",
      "INFO:tensorflow:Evaluation [59/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [61/100]\n",
      "INFO:tensorflow:Evaluation [62/100]\n",
      "INFO:tensorflow:Evaluation [63/100]\n",
      "INFO:tensorflow:Evaluation [64/100]\n",
      "INFO:tensorflow:Evaluation [65/100]\n",
      "INFO:tensorflow:Evaluation [66/100]\n",
      "INFO:tensorflow:Evaluation [67/100]\n",
      "INFO:tensorflow:Evaluation [68/100]\n",
      "INFO:tensorflow:Evaluation [69/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [71/100]\n",
      "INFO:tensorflow:Evaluation [72/100]\n",
      "INFO:tensorflow:Evaluation [73/100]\n",
      "INFO:tensorflow:Evaluation [74/100]\n",
      "INFO:tensorflow:Evaluation [75/100]\n",
      "INFO:tensorflow:Evaluation [76/100]\n",
      "INFO:tensorflow:Evaluation [77/100]\n",
      "INFO:tensorflow:Evaluation [78/100]\n",
      "INFO:tensorflow:Evaluation [79/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [81/100]\n",
      "INFO:tensorflow:Evaluation [82/100]\n",
      "INFO:tensorflow:Evaluation [83/100]\n",
      "INFO:tensorflow:Evaluation [84/100]\n",
      "INFO:tensorflow:Evaluation [85/100]\n",
      "INFO:tensorflow:Evaluation [86/100]\n",
      "INFO:tensorflow:Evaluation [87/100]\n",
      "INFO:tensorflow:Evaluation [88/100]\n",
      "INFO:tensorflow:Evaluation [89/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [91/100]\n",
      "INFO:tensorflow:Evaluation [92/100]\n",
      "INFO:tensorflow:Evaluation [93/100]\n",
      "INFO:tensorflow:Evaluation [94/100]\n",
      "INFO:tensorflow:Evaluation [95/100]\n",
      "INFO:tensorflow:Evaluation [96/100]\n",
      "INFO:tensorflow:Evaluation [97/100]\n",
      "INFO:tensorflow:Evaluation [98/100]\n",
      "INFO:tensorflow:Evaluation [99/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-01-14:16:51\n",
      "INFO:tensorflow:Saving dict for global step 202: accuracy = 0.7795, accuracy/baseline_label_mean = 0.209, accuracy/threshold_0.500000_mean = 0.7795, auc = 0.649993, auc_precision_recall = 0.335734, global_step = 202, labels/actual_label_mean = 0.209, labels/prediction_mean = 0.146127, loss = 1.64795, precision/positive_threshold_0.500000_mean = 0.417266, recall/positive_threshold_0.500000_mean = 0.138756\n",
      "INFO:tensorflow:Validation (step 201): loss = 1.64795, accuracy = 0.7795, labels/prediction_mean = 0.146127, labels/actual_label_mean = 0.209, accuracy/baseline_label_mean = 0.209, auc = 0.649993, auc_precision_recall = 0.335734, accuracy/threshold_0.500000_mean = 0.7795, precision/positive_threshold_0.500000_mean = 0.417266, recall/positive_threshold_0.500000_mean = 0.138756, global_step = 202\n",
      "INFO:tensorflow:Saving checkpoints for 302 into models/model_WIDE_AND_DEEP_1556720127\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 6.76205\n",
      "features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-01-14:17:02\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556720127\\model.ckpt-302\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Evaluation [39/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [41/100]\n",
      "INFO:tensorflow:Evaluation [42/100]\n",
      "INFO:tensorflow:Evaluation [43/100]\n",
      "INFO:tensorflow:Evaluation [44/100]\n",
      "INFO:tensorflow:Evaluation [45/100]\n",
      "INFO:tensorflow:Evaluation [46/100]\n",
      "INFO:tensorflow:Evaluation [47/100]\n",
      "INFO:tensorflow:Evaluation [48/100]\n",
      "INFO:tensorflow:Evaluation [49/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [51/100]\n",
      "INFO:tensorflow:Evaluation [52/100]\n",
      "INFO:tensorflow:Evaluation [53/100]\n",
      "INFO:tensorflow:Evaluation [54/100]\n",
      "INFO:tensorflow:Evaluation [55/100]\n",
      "INFO:tensorflow:Evaluation [56/100]\n",
      "INFO:tensorflow:Evaluation [57/100]\n",
      "INFO:tensorflow:Evaluation [58/100]\n",
      "INFO:tensorflow:Evaluation [59/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [61/100]\n",
      "INFO:tensorflow:Evaluation [62/100]\n",
      "INFO:tensorflow:Evaluation [63/100]\n",
      "INFO:tensorflow:Evaluation [64/100]\n",
      "INFO:tensorflow:Evaluation [65/100]\n",
      "INFO:tensorflow:Evaluation [66/100]\n",
      "INFO:tensorflow:Evaluation [67/100]\n",
      "INFO:tensorflow:Evaluation [68/100]\n",
      "INFO:tensorflow:Evaluation [69/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [71/100]\n",
      "INFO:tensorflow:Evaluation [72/100]\n",
      "INFO:tensorflow:Evaluation [73/100]\n",
      "INFO:tensorflow:Evaluation [74/100]\n",
      "INFO:tensorflow:Evaluation [75/100]\n",
      "INFO:tensorflow:Evaluation [76/100]\n",
      "INFO:tensorflow:Evaluation [77/100]\n",
      "INFO:tensorflow:Evaluation [78/100]\n",
      "INFO:tensorflow:Evaluation [79/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [81/100]\n",
      "INFO:tensorflow:Evaluation [82/100]\n",
      "INFO:tensorflow:Evaluation [83/100]\n",
      "INFO:tensorflow:Evaluation [84/100]\n",
      "INFO:tensorflow:Evaluation [85/100]\n",
      "INFO:tensorflow:Evaluation [86/100]\n",
      "INFO:tensorflow:Evaluation [87/100]\n",
      "INFO:tensorflow:Evaluation [88/100]\n",
      "INFO:tensorflow:Evaluation [89/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [91/100]\n",
      "INFO:tensorflow:Evaluation [92/100]\n",
      "INFO:tensorflow:Evaluation [93/100]\n",
      "INFO:tensorflow:Evaluation [94/100]\n",
      "INFO:tensorflow:Evaluation [95/100]\n",
      "INFO:tensorflow:Evaluation [96/100]\n",
      "INFO:tensorflow:Evaluation [97/100]\n",
      "INFO:tensorflow:Evaluation [98/100]\n",
      "INFO:tensorflow:Evaluation [99/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-01-14:17:06\n",
      "INFO:tensorflow:Saving dict for global step 302: accuracy = 0.568, accuracy/baseline_label_mean = 0.209, accuracy/threshold_0.500000_mean = 0.568, auc = 0.559431, auc_precision_recall = 0.218126, global_step = 302, labels/actual_label_mean = 0.209, labels/prediction_mean = 0.477377, loss = 1.50869, precision/positive_threshold_0.500000_mean = 0.248307, recall/positive_threshold_0.500000_mean = 0.526316\n",
      "INFO:tensorflow:Validation (step 301): loss = 1.50869, accuracy = 0.568, labels/prediction_mean = 0.477377, labels/actual_label_mean = 0.209, accuracy/baseline_label_mean = 0.209, auc = 0.559431, auc_precision_recall = 0.218126, accuracy/threshold_0.500000_mean = 0.568, precision/positive_threshold_0.500000_mean = 0.248307, recall/positive_threshold_0.500000_mean = 0.526316, global_step = 302\n",
      "INFO:tensorflow:Saving checkpoints for 402 into models/model_WIDE_AND_DEEP_1556720127\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 6.77988\n",
      "INFO:tensorflow:loss = 0.488683, step = 402 (29.538 sec)\n",
      "features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-01-14:17:18\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556720127\\model.ckpt-402\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Evaluation [39/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [41/100]\n",
      "INFO:tensorflow:Evaluation [42/100]\n",
      "INFO:tensorflow:Evaluation [43/100]\n",
      "INFO:tensorflow:Evaluation [44/100]\n",
      "INFO:tensorflow:Evaluation [45/100]\n",
      "INFO:tensorflow:Evaluation [46/100]\n",
      "INFO:tensorflow:Evaluation [47/100]\n",
      "INFO:tensorflow:Evaluation [48/100]\n",
      "INFO:tensorflow:Evaluation [49/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [51/100]\n",
      "INFO:tensorflow:Evaluation [52/100]\n",
      "INFO:tensorflow:Evaluation [53/100]\n",
      "INFO:tensorflow:Evaluation [54/100]\n",
      "INFO:tensorflow:Evaluation [55/100]\n",
      "INFO:tensorflow:Evaluation [56/100]\n",
      "INFO:tensorflow:Evaluation [57/100]\n",
      "INFO:tensorflow:Evaluation [58/100]\n",
      "INFO:tensorflow:Evaluation [59/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [61/100]\n",
      "INFO:tensorflow:Evaluation [62/100]\n",
      "INFO:tensorflow:Evaluation [63/100]\n",
      "INFO:tensorflow:Evaluation [64/100]\n",
      "INFO:tensorflow:Evaluation [65/100]\n",
      "INFO:tensorflow:Evaluation [66/100]\n",
      "INFO:tensorflow:Evaluation [67/100]\n",
      "INFO:tensorflow:Evaluation [68/100]\n",
      "INFO:tensorflow:Evaluation [69/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [71/100]\n",
      "INFO:tensorflow:Evaluation [72/100]\n",
      "INFO:tensorflow:Evaluation [73/100]\n",
      "INFO:tensorflow:Evaluation [74/100]\n",
      "INFO:tensorflow:Evaluation [75/100]\n",
      "INFO:tensorflow:Evaluation [76/100]\n",
      "INFO:tensorflow:Evaluation [77/100]\n",
      "INFO:tensorflow:Evaluation [78/100]\n",
      "INFO:tensorflow:Evaluation [79/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [81/100]\n",
      "INFO:tensorflow:Evaluation [82/100]\n",
      "INFO:tensorflow:Evaluation [83/100]\n",
      "INFO:tensorflow:Evaluation [84/100]\n",
      "INFO:tensorflow:Evaluation [85/100]\n",
      "INFO:tensorflow:Evaluation [86/100]\n",
      "INFO:tensorflow:Evaluation [87/100]\n",
      "INFO:tensorflow:Evaluation [88/100]\n",
      "INFO:tensorflow:Evaluation [89/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [91/100]\n",
      "INFO:tensorflow:Evaluation [92/100]\n",
      "INFO:tensorflow:Evaluation [93/100]\n",
      "INFO:tensorflow:Evaluation [94/100]\n",
      "INFO:tensorflow:Evaluation [95/100]\n",
      "INFO:tensorflow:Evaluation [96/100]\n",
      "INFO:tensorflow:Evaluation [97/100]\n",
      "INFO:tensorflow:Evaluation [98/100]\n",
      "INFO:tensorflow:Evaluation [99/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-01-14:17:21\n",
      "INFO:tensorflow:Saving dict for global step 402: accuracy = 0.79, accuracy/baseline_label_mean = 0.209, accuracy/threshold_0.500000_mean = 0.79, auc = 0.700955, auc_precision_recall = 0.377003, global_step = 402, labels/actual_label_mean = 0.209, labels/prediction_mean = 0.204433, loss = 0.557425, precision/positive_threshold_0.500000_mean = 0.492063, recall/positive_threshold_0.500000_mean = 0.148325\n",
      "INFO:tensorflow:Validation (step 401): loss = 0.557425, accuracy = 0.79, labels/prediction_mean = 0.204433, labels/actual_label_mean = 0.209, accuracy/baseline_label_mean = 0.209, auc = 0.700955, auc_precision_recall = 0.377003, accuracy/threshold_0.500000_mean = 0.79, precision/positive_threshold_0.500000_mean = 0.492063, recall/positive_threshold_0.500000_mean = 0.148325, global_step = 402\n",
      "INFO:tensorflow:Saving checkpoints for 502 into models/model_WIDE_AND_DEEP_1556720127\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 6.57405\n",
      "features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-01-14:17:32\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556720127\\model.ckpt-502\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Evaluation [39/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [41/100]\n",
      "INFO:tensorflow:Evaluation [42/100]\n",
      "INFO:tensorflow:Evaluation [43/100]\n",
      "INFO:tensorflow:Evaluation [44/100]\n",
      "INFO:tensorflow:Evaluation [45/100]\n",
      "INFO:tensorflow:Evaluation [46/100]\n",
      "INFO:tensorflow:Evaluation [47/100]\n",
      "INFO:tensorflow:Evaluation [48/100]\n",
      "INFO:tensorflow:Evaluation [49/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [51/100]\n",
      "INFO:tensorflow:Evaluation [52/100]\n",
      "INFO:tensorflow:Evaluation [53/100]\n",
      "INFO:tensorflow:Evaluation [54/100]\n",
      "INFO:tensorflow:Evaluation [55/100]\n",
      "INFO:tensorflow:Evaluation [56/100]\n",
      "INFO:tensorflow:Evaluation [57/100]\n",
      "INFO:tensorflow:Evaluation [58/100]\n",
      "INFO:tensorflow:Evaluation [59/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [61/100]\n",
      "INFO:tensorflow:Evaluation [62/100]\n",
      "INFO:tensorflow:Evaluation [63/100]\n",
      "INFO:tensorflow:Evaluation [64/100]\n",
      "INFO:tensorflow:Evaluation [65/100]\n",
      "INFO:tensorflow:Evaluation [66/100]\n",
      "INFO:tensorflow:Evaluation [67/100]\n",
      "INFO:tensorflow:Evaluation [68/100]\n",
      "INFO:tensorflow:Evaluation [69/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [71/100]\n",
      "INFO:tensorflow:Evaluation [72/100]\n",
      "INFO:tensorflow:Evaluation [73/100]\n",
      "INFO:tensorflow:Evaluation [74/100]\n",
      "INFO:tensorflow:Evaluation [75/100]\n",
      "INFO:tensorflow:Evaluation [76/100]\n",
      "INFO:tensorflow:Evaluation [77/100]\n",
      "INFO:tensorflow:Evaluation [78/100]\n",
      "INFO:tensorflow:Evaluation [79/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [81/100]\n",
      "INFO:tensorflow:Evaluation [82/100]\n",
      "INFO:tensorflow:Evaluation [83/100]\n",
      "INFO:tensorflow:Evaluation [84/100]\n",
      "INFO:tensorflow:Evaluation [85/100]\n",
      "INFO:tensorflow:Evaluation [86/100]\n",
      "INFO:tensorflow:Evaluation [87/100]\n",
      "INFO:tensorflow:Evaluation [88/100]\n",
      "INFO:tensorflow:Evaluation [89/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [91/100]\n",
      "INFO:tensorflow:Evaluation [92/100]\n",
      "INFO:tensorflow:Evaluation [93/100]\n",
      "INFO:tensorflow:Evaluation [94/100]\n",
      "INFO:tensorflow:Evaluation [95/100]\n",
      "INFO:tensorflow:Evaluation [96/100]\n",
      "INFO:tensorflow:Evaluation [97/100]\n",
      "INFO:tensorflow:Evaluation [98/100]\n",
      "INFO:tensorflow:Evaluation [99/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-01-14:17:36\n",
      "INFO:tensorflow:Saving dict for global step 502: accuracy = 0.767, accuracy/baseline_label_mean = 0.209, accuracy/threshold_0.500000_mean = 0.767, auc = 0.680741, auc_precision_recall = 0.320326, global_step = 502, labels/actual_label_mean = 0.209, labels/prediction_mean = 0.279146, loss = 0.516375, precision/positive_threshold_0.500000_mean = 0.389908, recall/positive_threshold_0.500000_mean = 0.203349\n",
      "INFO:tensorflow:Validation (step 501): loss = 0.516375, accuracy = 0.767, labels/prediction_mean = 0.279146, labels/actual_label_mean = 0.209, accuracy/baseline_label_mean = 0.209, auc = 0.680741, auc_precision_recall = 0.320326, accuracy/threshold_0.500000_mean = 0.767, precision/positive_threshold_0.500000_mean = 0.389908, recall/positive_threshold_0.500000_mean = 0.203349, global_step = 502\n",
      "INFO:tensorflow:Saving checkpoints for 602 into models/model_WIDE_AND_DEEP_1556720127\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 6.66491\n",
      "INFO:tensorflow:loss = 0.478766, step = 602 (30.216 sec)\n",
      "features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-01-14:17:47\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556720127\\model.ckpt-602\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Evaluation [39/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [41/100]\n",
      "INFO:tensorflow:Evaluation [42/100]\n",
      "INFO:tensorflow:Evaluation [43/100]\n",
      "INFO:tensorflow:Evaluation [44/100]\n",
      "INFO:tensorflow:Evaluation [45/100]\n",
      "INFO:tensorflow:Evaluation [46/100]\n",
      "INFO:tensorflow:Evaluation [47/100]\n",
      "INFO:tensorflow:Evaluation [48/100]\n",
      "INFO:tensorflow:Evaluation [49/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [51/100]\n",
      "INFO:tensorflow:Evaluation [52/100]\n",
      "INFO:tensorflow:Evaluation [53/100]\n",
      "INFO:tensorflow:Evaluation [54/100]\n",
      "INFO:tensorflow:Evaluation [55/100]\n",
      "INFO:tensorflow:Evaluation [56/100]\n",
      "INFO:tensorflow:Evaluation [57/100]\n",
      "INFO:tensorflow:Evaluation [58/100]\n",
      "INFO:tensorflow:Evaluation [59/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [61/100]\n",
      "INFO:tensorflow:Evaluation [62/100]\n",
      "INFO:tensorflow:Evaluation [63/100]\n",
      "INFO:tensorflow:Evaluation [64/100]\n",
      "INFO:tensorflow:Evaluation [65/100]\n",
      "INFO:tensorflow:Evaluation [66/100]\n",
      "INFO:tensorflow:Evaluation [67/100]\n",
      "INFO:tensorflow:Evaluation [68/100]\n",
      "INFO:tensorflow:Evaluation [69/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [71/100]\n",
      "INFO:tensorflow:Evaluation [72/100]\n",
      "INFO:tensorflow:Evaluation [73/100]\n",
      "INFO:tensorflow:Evaluation [74/100]\n",
      "INFO:tensorflow:Evaluation [75/100]\n",
      "INFO:tensorflow:Evaluation [76/100]\n",
      "INFO:tensorflow:Evaluation [77/100]\n",
      "INFO:tensorflow:Evaluation [78/100]\n",
      "INFO:tensorflow:Evaluation [79/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [81/100]\n",
      "INFO:tensorflow:Evaluation [82/100]\n",
      "INFO:tensorflow:Evaluation [83/100]\n",
      "INFO:tensorflow:Evaluation [84/100]\n",
      "INFO:tensorflow:Evaluation [85/100]\n",
      "INFO:tensorflow:Evaluation [86/100]\n",
      "INFO:tensorflow:Evaluation [87/100]\n",
      "INFO:tensorflow:Evaluation [88/100]\n",
      "INFO:tensorflow:Evaluation [89/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [91/100]\n",
      "INFO:tensorflow:Evaluation [92/100]\n",
      "INFO:tensorflow:Evaluation [93/100]\n",
      "INFO:tensorflow:Evaluation [94/100]\n",
      "INFO:tensorflow:Evaluation [95/100]\n",
      "INFO:tensorflow:Evaluation [96/100]\n",
      "INFO:tensorflow:Evaluation [97/100]\n",
      "INFO:tensorflow:Evaluation [98/100]\n",
      "INFO:tensorflow:Evaluation [99/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-01-14:17:51\n",
      "INFO:tensorflow:Saving dict for global step 602: accuracy = 0.791, accuracy/baseline_label_mean = 0.209, accuracy/threshold_0.500000_mean = 0.791, auc = 0.699903, auc_precision_recall = 0.37707, global_step = 602, labels/actual_label_mean = 0.209, labels/prediction_mean = 0.183832, loss = 0.591904, precision/positive_threshold_0.500000_mean = 0.5, recall/positive_threshold_0.500000_mean = 0.136364\n",
      "INFO:tensorflow:Validation (step 601): loss = 0.591904, accuracy = 0.791, labels/prediction_mean = 0.183832, labels/actual_label_mean = 0.209, accuracy/baseline_label_mean = 0.209, auc = 0.699903, auc_precision_recall = 0.37707, accuracy/threshold_0.500000_mean = 0.791, precision/positive_threshold_0.500000_mean = 0.5, recall/positive_threshold_0.500000_mean = 0.136364, global_step = 602\n",
      "INFO:tensorflow:Saving checkpoints for 702 into models/model_WIDE_AND_DEEP_1556720127\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 6.84187\n",
      "features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-01-14:18:02\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556720127\\model.ckpt-702\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Evaluation [39/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [41/100]\n",
      "INFO:tensorflow:Evaluation [42/100]\n",
      "INFO:tensorflow:Evaluation [43/100]\n",
      "INFO:tensorflow:Evaluation [44/100]\n",
      "INFO:tensorflow:Evaluation [45/100]\n",
      "INFO:tensorflow:Evaluation [46/100]\n",
      "INFO:tensorflow:Evaluation [47/100]\n",
      "INFO:tensorflow:Evaluation [48/100]\n",
      "INFO:tensorflow:Evaluation [49/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [51/100]\n",
      "INFO:tensorflow:Evaluation [52/100]\n",
      "INFO:tensorflow:Evaluation [53/100]\n",
      "INFO:tensorflow:Evaluation [54/100]\n",
      "INFO:tensorflow:Evaluation [55/100]\n",
      "INFO:tensorflow:Evaluation [56/100]\n",
      "INFO:tensorflow:Evaluation [57/100]\n",
      "INFO:tensorflow:Evaluation [58/100]\n",
      "INFO:tensorflow:Evaluation [59/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [61/100]\n",
      "INFO:tensorflow:Evaluation [62/100]\n",
      "INFO:tensorflow:Evaluation [63/100]\n",
      "INFO:tensorflow:Evaluation [64/100]\n",
      "INFO:tensorflow:Evaluation [65/100]\n",
      "INFO:tensorflow:Evaluation [66/100]\n",
      "INFO:tensorflow:Evaluation [67/100]\n",
      "INFO:tensorflow:Evaluation [68/100]\n",
      "INFO:tensorflow:Evaluation [69/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [71/100]\n",
      "INFO:tensorflow:Evaluation [72/100]\n",
      "INFO:tensorflow:Evaluation [73/100]\n",
      "INFO:tensorflow:Evaluation [74/100]\n",
      "INFO:tensorflow:Evaluation [75/100]\n",
      "INFO:tensorflow:Evaluation [76/100]\n",
      "INFO:tensorflow:Evaluation [77/100]\n",
      "INFO:tensorflow:Evaluation [78/100]\n",
      "INFO:tensorflow:Evaluation [79/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [81/100]\n",
      "INFO:tensorflow:Evaluation [82/100]\n",
      "INFO:tensorflow:Evaluation [83/100]\n",
      "INFO:tensorflow:Evaluation [84/100]\n",
      "INFO:tensorflow:Evaluation [85/100]\n",
      "INFO:tensorflow:Evaluation [86/100]\n",
      "INFO:tensorflow:Evaluation [87/100]\n",
      "INFO:tensorflow:Evaluation [88/100]\n",
      "INFO:tensorflow:Evaluation [89/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [91/100]\n",
      "INFO:tensorflow:Evaluation [92/100]\n",
      "INFO:tensorflow:Evaluation [93/100]\n",
      "INFO:tensorflow:Evaluation [94/100]\n",
      "INFO:tensorflow:Evaluation [95/100]\n",
      "INFO:tensorflow:Evaluation [96/100]\n",
      "INFO:tensorflow:Evaluation [97/100]\n",
      "INFO:tensorflow:Evaluation [98/100]\n",
      "INFO:tensorflow:Evaluation [99/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-01-14:18:06\n",
      "INFO:tensorflow:Saving dict for global step 702: accuracy = 0.7855, accuracy/baseline_label_mean = 0.209, accuracy/threshold_0.500000_mean = 0.7855, auc = 0.708631, auc_precision_recall = 0.373972, global_step = 702, labels/actual_label_mean = 0.209, labels/prediction_mean = 0.262323, loss = 0.479515, precision/positive_threshold_0.500000_mean = 0.465839, recall/positive_threshold_0.500000_mean = 0.179426\n",
      "INFO:tensorflow:Validation (step 701): loss = 0.479515, accuracy = 0.7855, labels/prediction_mean = 0.262323, labels/actual_label_mean = 0.209, accuracy/baseline_label_mean = 0.209, auc = 0.708631, auc_precision_recall = 0.373972, accuracy/threshold_0.500000_mean = 0.7855, precision/positive_threshold_0.500000_mean = 0.465839, recall/positive_threshold_0.500000_mean = 0.179426, global_step = 702\n",
      "INFO:tensorflow:Saving checkpoints for 802 into models/model_WIDE_AND_DEEP_1556720127\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 6.52613\n",
      "INFO:tensorflow:loss = 0.395205, step = 802 (29.939 sec)\n",
      "features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-01-14:18:18\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556720127\\model.ckpt-802\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Evaluation [39/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [41/100]\n",
      "INFO:tensorflow:Evaluation [42/100]\n",
      "INFO:tensorflow:Evaluation [43/100]\n",
      "INFO:tensorflow:Evaluation [44/100]\n",
      "INFO:tensorflow:Evaluation [45/100]\n",
      "INFO:tensorflow:Evaluation [46/100]\n",
      "INFO:tensorflow:Evaluation [47/100]\n",
      "INFO:tensorflow:Evaluation [48/100]\n",
      "INFO:tensorflow:Evaluation [49/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [51/100]\n",
      "INFO:tensorflow:Evaluation [52/100]\n",
      "INFO:tensorflow:Evaluation [53/100]\n",
      "INFO:tensorflow:Evaluation [54/100]\n",
      "INFO:tensorflow:Evaluation [55/100]\n",
      "INFO:tensorflow:Evaluation [56/100]\n",
      "INFO:tensorflow:Evaluation [57/100]\n",
      "INFO:tensorflow:Evaluation [58/100]\n",
      "INFO:tensorflow:Evaluation [59/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [61/100]\n",
      "INFO:tensorflow:Evaluation [62/100]\n",
      "INFO:tensorflow:Evaluation [63/100]\n",
      "INFO:tensorflow:Evaluation [64/100]\n",
      "INFO:tensorflow:Evaluation [65/100]\n",
      "INFO:tensorflow:Evaluation [66/100]\n",
      "INFO:tensorflow:Evaluation [67/100]\n",
      "INFO:tensorflow:Evaluation [68/100]\n",
      "INFO:tensorflow:Evaluation [69/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [71/100]\n",
      "INFO:tensorflow:Evaluation [72/100]\n",
      "INFO:tensorflow:Evaluation [73/100]\n",
      "INFO:tensorflow:Evaluation [74/100]\n",
      "INFO:tensorflow:Evaluation [75/100]\n",
      "INFO:tensorflow:Evaluation [76/100]\n",
      "INFO:tensorflow:Evaluation [77/100]\n",
      "INFO:tensorflow:Evaluation [78/100]\n",
      "INFO:tensorflow:Evaluation [79/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [81/100]\n",
      "INFO:tensorflow:Evaluation [82/100]\n",
      "INFO:tensorflow:Evaluation [83/100]\n",
      "INFO:tensorflow:Evaluation [84/100]\n",
      "INFO:tensorflow:Evaluation [85/100]\n",
      "INFO:tensorflow:Evaluation [86/100]\n",
      "INFO:tensorflow:Evaluation [87/100]\n",
      "INFO:tensorflow:Evaluation [88/100]\n",
      "INFO:tensorflow:Evaluation [89/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [91/100]\n",
      "INFO:tensorflow:Evaluation [92/100]\n",
      "INFO:tensorflow:Evaluation [93/100]\n",
      "INFO:tensorflow:Evaluation [94/100]\n",
      "INFO:tensorflow:Evaluation [95/100]\n",
      "INFO:tensorflow:Evaluation [96/100]\n",
      "INFO:tensorflow:Evaluation [97/100]\n",
      "INFO:tensorflow:Evaluation [98/100]\n",
      "INFO:tensorflow:Evaluation [99/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-01-14:18:21\n",
      "INFO:tensorflow:Saving dict for global step 802: accuracy = 0.7895, accuracy/baseline_label_mean = 0.209, accuracy/threshold_0.500000_mean = 0.7895, auc = 0.722598, auc_precision_recall = 0.388464, global_step = 802, labels/actual_label_mean = 0.209, labels/prediction_mean = 0.200899, loss = 0.469193, precision/positive_threshold_0.500000_mean = 0.483871, recall/positive_threshold_0.500000_mean = 0.107656\n",
      "INFO:tensorflow:Validation (step 801): loss = 0.469193, accuracy = 0.7895, labels/prediction_mean = 0.200899, labels/actual_label_mean = 0.209, accuracy/baseline_label_mean = 0.209, auc = 0.722598, auc_precision_recall = 0.388464, accuracy/threshold_0.500000_mean = 0.7895, precision/positive_threshold_0.500000_mean = 0.483871, recall/positive_threshold_0.500000_mean = 0.107656, global_step = 802\n",
      "INFO:tensorflow:Saving checkpoints for 902 into models/model_WIDE_AND_DEEP_1556720127\\model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 6.36699\n",
      "features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-01-14:18:33\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556720127\\model.ckpt-902\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Evaluation [39/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [41/100]\n",
      "INFO:tensorflow:Evaluation [42/100]\n",
      "INFO:tensorflow:Evaluation [43/100]\n",
      "INFO:tensorflow:Evaluation [44/100]\n",
      "INFO:tensorflow:Evaluation [45/100]\n",
      "INFO:tensorflow:Evaluation [46/100]\n",
      "INFO:tensorflow:Evaluation [47/100]\n",
      "INFO:tensorflow:Evaluation [48/100]\n",
      "INFO:tensorflow:Evaluation [49/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [51/100]\n",
      "INFO:tensorflow:Evaluation [52/100]\n",
      "INFO:tensorflow:Evaluation [53/100]\n",
      "INFO:tensorflow:Evaluation [54/100]\n",
      "INFO:tensorflow:Evaluation [55/100]\n",
      "INFO:tensorflow:Evaluation [56/100]\n",
      "INFO:tensorflow:Evaluation [57/100]\n",
      "INFO:tensorflow:Evaluation [58/100]\n",
      "INFO:tensorflow:Evaluation [59/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [61/100]\n",
      "INFO:tensorflow:Evaluation [62/100]\n",
      "INFO:tensorflow:Evaluation [63/100]\n",
      "INFO:tensorflow:Evaluation [64/100]\n",
      "INFO:tensorflow:Evaluation [65/100]\n",
      "INFO:tensorflow:Evaluation [66/100]\n",
      "INFO:tensorflow:Evaluation [67/100]\n",
      "INFO:tensorflow:Evaluation [68/100]\n",
      "INFO:tensorflow:Evaluation [69/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [71/100]\n",
      "INFO:tensorflow:Evaluation [72/100]\n",
      "INFO:tensorflow:Evaluation [73/100]\n",
      "INFO:tensorflow:Evaluation [74/100]\n",
      "INFO:tensorflow:Evaluation [75/100]\n",
      "INFO:tensorflow:Evaluation [76/100]\n",
      "INFO:tensorflow:Evaluation [77/100]\n",
      "INFO:tensorflow:Evaluation [78/100]\n",
      "INFO:tensorflow:Evaluation [79/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [81/100]\n",
      "INFO:tensorflow:Evaluation [82/100]\n",
      "INFO:tensorflow:Evaluation [83/100]\n",
      "INFO:tensorflow:Evaluation [84/100]\n",
      "INFO:tensorflow:Evaluation [85/100]\n",
      "INFO:tensorflow:Evaluation [86/100]\n",
      "INFO:tensorflow:Evaluation [87/100]\n",
      "INFO:tensorflow:Evaluation [88/100]\n",
      "INFO:tensorflow:Evaluation [89/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [91/100]\n",
      "INFO:tensorflow:Evaluation [92/100]\n",
      "INFO:tensorflow:Evaluation [93/100]\n",
      "INFO:tensorflow:Evaluation [94/100]\n",
      "INFO:tensorflow:Evaluation [95/100]\n",
      "INFO:tensorflow:Evaluation [96/100]\n",
      "INFO:tensorflow:Evaluation [97/100]\n",
      "INFO:tensorflow:Evaluation [98/100]\n",
      "INFO:tensorflow:Evaluation [99/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-01-14:18:37\n",
      "INFO:tensorflow:Saving dict for global step 902: accuracy = 0.7875, accuracy/baseline_label_mean = 0.209, accuracy/threshold_0.500000_mean = 0.7875, auc = 0.724809, auc_precision_recall = 0.384287, global_step = 902, labels/actual_label_mean = 0.209, labels/prediction_mean = 0.20923, loss = 0.465275, precision/positive_threshold_0.500000_mean = 0.466667, recall/positive_threshold_0.500000_mean = 0.117225\n",
      "INFO:tensorflow:Validation (step 901): loss = 0.465275, accuracy = 0.7875, labels/prediction_mean = 0.20923, labels/actual_label_mean = 0.209, accuracy/baseline_label_mean = 0.209, auc = 0.724809, auc_precision_recall = 0.384287, accuracy/threshold_0.500000_mean = 0.7875, precision/positive_threshold_0.500000_mean = 0.466667, recall/positive_threshold_0.500000_mean = 0.117225, global_step = 902\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into models/model_WIDE_AND_DEEP_1556720127\\model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.419244.\n",
      "features:{'I1': <tf.Tensor 'DecodeCSV:1' shape=(?,) dtype=int32>, 'I2': <tf.Tensor 'DecodeCSV:2' shape=(?,) dtype=int32>, 'I3': <tf.Tensor 'DecodeCSV:3' shape=(?,) dtype=int32>, 'I4': <tf.Tensor 'DecodeCSV:4' shape=(?,) dtype=int32>, 'I5': <tf.Tensor 'DecodeCSV:5' shape=(?,) dtype=int32>, 'I6': <tf.Tensor 'DecodeCSV:6' shape=(?,) dtype=int32>, 'I7': <tf.Tensor 'DecodeCSV:7' shape=(?,) dtype=int32>, 'I8': <tf.Tensor 'DecodeCSV:8' shape=(?,) dtype=int32>, 'I9': <tf.Tensor 'DecodeCSV:9' shape=(?,) dtype=int32>, 'I10': <tf.Tensor 'DecodeCSV:10' shape=(?,) dtype=int32>, 'I11': <tf.Tensor 'DecodeCSV:11' shape=(?,) dtype=int32>, 'I12': <tf.Tensor 'DecodeCSV:12' shape=(?,) dtype=int32>, 'I13': <tf.Tensor 'DecodeCSV:13' shape=(?,) dtype=int32>, 'C1': <tf.Tensor 'DecodeCSV:14' shape=(?,) dtype=string>, 'C2': <tf.Tensor 'DecodeCSV:15' shape=(?,) dtype=string>, 'C3': <tf.Tensor 'DecodeCSV:16' shape=(?,) dtype=string>, 'C4': <tf.Tensor 'DecodeCSV:17' shape=(?,) dtype=string>, 'C5': <tf.Tensor 'DecodeCSV:18' shape=(?,) dtype=string>, 'C6': <tf.Tensor 'DecodeCSV:19' shape=(?,) dtype=string>, 'C7': <tf.Tensor 'DecodeCSV:20' shape=(?,) dtype=string>, 'C8': <tf.Tensor 'DecodeCSV:21' shape=(?,) dtype=string>, 'C9': <tf.Tensor 'DecodeCSV:22' shape=(?,) dtype=string>, 'C10': <tf.Tensor 'DecodeCSV:23' shape=(?,) dtype=string>, 'C11': <tf.Tensor 'DecodeCSV:24' shape=(?,) dtype=string>, 'C12': <tf.Tensor 'DecodeCSV:25' shape=(?,) dtype=string>, 'C13': <tf.Tensor 'DecodeCSV:26' shape=(?,) dtype=string>, 'C14': <tf.Tensor 'DecodeCSV:27' shape=(?,) dtype=string>, 'C15': <tf.Tensor 'DecodeCSV:28' shape=(?,) dtype=string>, 'C16': <tf.Tensor 'DecodeCSV:29' shape=(?,) dtype=string>, 'C17': <tf.Tensor 'DecodeCSV:30' shape=(?,) dtype=string>, 'C18': <tf.Tensor 'DecodeCSV:31' shape=(?,) dtype=string>, 'C19': <tf.Tensor 'DecodeCSV:32' shape=(?,) dtype=string>, 'C20': <tf.Tensor 'DecodeCSV:33' shape=(?,) dtype=string>, 'C21': <tf.Tensor 'DecodeCSV:34' shape=(?,) dtype=string>, 'C22': <tf.Tensor 'DecodeCSV:35' shape=(?,) dtype=string>, 'C23': <tf.Tensor 'DecodeCSV:36' shape=(?,) dtype=string>, 'C24': <tf.Tensor 'DecodeCSV:37' shape=(?,) dtype=string>, 'C25': <tf.Tensor 'DecodeCSV:38' shape=(?,) dtype=string>, 'C26': <tf.Tensor 'DecodeCSV:39' shape=(?,) dtype=string>}\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:Rank of input Tensor (1) should be the same as output_rank (2) for column. Will attempt to expand dims. It is highly recommended that you resize your input, as this behavior may change.\n",
      "WARNING:tensorflow:From D:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\head.py:625: scalar_summary (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2016-11-30.\n",
      "Instructions for updating:\n",
      "Please switch to tf.summary.scalar. Note that tf.summary.scalar uses the node name instead of the tag. This means that TensorFlow will automatically de-duplicate summary names based on the scope they are created in. Also, passing a tensor or list of tags to a scalar summary op is no longer supported.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-01-14:18:49\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556720127\\model.ckpt-1000\n",
      "INFO:tensorflow:Evaluation [1/100]\n",
      "INFO:tensorflow:Evaluation [2/100]\n",
      "INFO:tensorflow:Evaluation [3/100]\n",
      "INFO:tensorflow:Evaluation [4/100]\n",
      "INFO:tensorflow:Evaluation [5/100]\n",
      "INFO:tensorflow:Evaluation [6/100]\n",
      "INFO:tensorflow:Evaluation [7/100]\n",
      "INFO:tensorflow:Evaluation [8/100]\n",
      "INFO:tensorflow:Evaluation [9/100]\n",
      "INFO:tensorflow:Evaluation [10/100]\n",
      "INFO:tensorflow:Evaluation [11/100]\n",
      "INFO:tensorflow:Evaluation [12/100]\n",
      "INFO:tensorflow:Evaluation [13/100]\n",
      "INFO:tensorflow:Evaluation [14/100]\n",
      "INFO:tensorflow:Evaluation [15/100]\n",
      "INFO:tensorflow:Evaluation [16/100]\n",
      "INFO:tensorflow:Evaluation [17/100]\n",
      "INFO:tensorflow:Evaluation [18/100]\n",
      "INFO:tensorflow:Evaluation [19/100]\n",
      "INFO:tensorflow:Evaluation [20/100]\n",
      "INFO:tensorflow:Evaluation [21/100]\n",
      "INFO:tensorflow:Evaluation [22/100]\n",
      "INFO:tensorflow:Evaluation [23/100]\n",
      "INFO:tensorflow:Evaluation [24/100]\n",
      "INFO:tensorflow:Evaluation [25/100]\n",
      "INFO:tensorflow:Evaluation [26/100]\n",
      "INFO:tensorflow:Evaluation [27/100]\n",
      "INFO:tensorflow:Evaluation [28/100]\n",
      "INFO:tensorflow:Evaluation [29/100]\n",
      "INFO:tensorflow:Evaluation [30/100]\n",
      "INFO:tensorflow:Evaluation [31/100]\n",
      "INFO:tensorflow:Evaluation [32/100]\n",
      "INFO:tensorflow:Evaluation [33/100]\n",
      "INFO:tensorflow:Evaluation [34/100]\n",
      "INFO:tensorflow:Evaluation [35/100]\n",
      "INFO:tensorflow:Evaluation [36/100]\n",
      "INFO:tensorflow:Evaluation [37/100]\n",
      "INFO:tensorflow:Evaluation [38/100]\n",
      "INFO:tensorflow:Evaluation [39/100]\n",
      "INFO:tensorflow:Evaluation [40/100]\n",
      "INFO:tensorflow:Evaluation [41/100]\n",
      "INFO:tensorflow:Evaluation [42/100]\n",
      "INFO:tensorflow:Evaluation [43/100]\n",
      "INFO:tensorflow:Evaluation [44/100]\n",
      "INFO:tensorflow:Evaluation [45/100]\n",
      "INFO:tensorflow:Evaluation [46/100]\n",
      "INFO:tensorflow:Evaluation [47/100]\n",
      "INFO:tensorflow:Evaluation [48/100]\n",
      "INFO:tensorflow:Evaluation [49/100]\n",
      "INFO:tensorflow:Evaluation [50/100]\n",
      "INFO:tensorflow:Evaluation [51/100]\n",
      "INFO:tensorflow:Evaluation [52/100]\n",
      "INFO:tensorflow:Evaluation [53/100]\n",
      "INFO:tensorflow:Evaluation [54/100]\n",
      "INFO:tensorflow:Evaluation [55/100]\n",
      "INFO:tensorflow:Evaluation [56/100]\n",
      "INFO:tensorflow:Evaluation [57/100]\n",
      "INFO:tensorflow:Evaluation [58/100]\n",
      "INFO:tensorflow:Evaluation [59/100]\n",
      "INFO:tensorflow:Evaluation [60/100]\n",
      "INFO:tensorflow:Evaluation [61/100]\n",
      "INFO:tensorflow:Evaluation [62/100]\n",
      "INFO:tensorflow:Evaluation [63/100]\n",
      "INFO:tensorflow:Evaluation [64/100]\n",
      "INFO:tensorflow:Evaluation [65/100]\n",
      "INFO:tensorflow:Evaluation [66/100]\n",
      "INFO:tensorflow:Evaluation [67/100]\n",
      "INFO:tensorflow:Evaluation [68/100]\n",
      "INFO:tensorflow:Evaluation [69/100]\n",
      "INFO:tensorflow:Evaluation [70/100]\n",
      "INFO:tensorflow:Evaluation [71/100]\n",
      "INFO:tensorflow:Evaluation [72/100]\n",
      "INFO:tensorflow:Evaluation [73/100]\n",
      "INFO:tensorflow:Evaluation [74/100]\n",
      "INFO:tensorflow:Evaluation [75/100]\n",
      "INFO:tensorflow:Evaluation [76/100]\n",
      "INFO:tensorflow:Evaluation [77/100]\n",
      "INFO:tensorflow:Evaluation [78/100]\n",
      "INFO:tensorflow:Evaluation [79/100]\n",
      "INFO:tensorflow:Evaluation [80/100]\n",
      "INFO:tensorflow:Evaluation [81/100]\n",
      "INFO:tensorflow:Evaluation [82/100]\n",
      "INFO:tensorflow:Evaluation [83/100]\n",
      "INFO:tensorflow:Evaluation [84/100]\n",
      "INFO:tensorflow:Evaluation [85/100]\n",
      "INFO:tensorflow:Evaluation [86/100]\n",
      "INFO:tensorflow:Evaluation [87/100]\n",
      "INFO:tensorflow:Evaluation [88/100]\n",
      "INFO:tensorflow:Evaluation [89/100]\n",
      "INFO:tensorflow:Evaluation [90/100]\n",
      "INFO:tensorflow:Evaluation [91/100]\n",
      "INFO:tensorflow:Evaluation [92/100]\n",
      "INFO:tensorflow:Evaluation [93/100]\n",
      "INFO:tensorflow:Evaluation [94/100]\n",
      "INFO:tensorflow:Evaluation [95/100]\n",
      "INFO:tensorflow:Evaluation [96/100]\n",
      "INFO:tensorflow:Evaluation [97/100]\n",
      "INFO:tensorflow:Evaluation [98/100]\n",
      "INFO:tensorflow:Evaluation [99/100]\n",
      "INFO:tensorflow:Evaluation [100/100]\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-01-14:18:53\n",
      "INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.792, accuracy/baseline_label_mean = 0.209, accuracy/threshold_0.500000_mean = 0.792, auc = 0.715159, auc_precision_recall = 0.389059, global_step = 1000, labels/actual_label_mean = 0.209, labels/prediction_mean = 0.183448, loss = 0.495491, precision/positive_threshold_0.500000_mean = 0.51087, recall/positive_threshold_0.500000_mean = 0.11244\n",
      "INFO:tensorflow:Restoring parameters from models/model_WIDE_AND_DEEP_1556720127\\model.ckpt-1000\n"
     ]
    },
    {
     "ename": "NotFoundError",
     "evalue": "Failed to create a directory: models/model_WIDE_AND_DEEP_1556720127\\export\\Servo\\1556720345",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNotFoundError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-201abc2ecd9e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mmodel_dir\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mcreate_model_dir\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmodel_type\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mMODEL_TYPE\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mmetrics\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0moutput_folder\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlearn_runner\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mexperiment_fn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmodel_dir\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'Accuracy: {}'\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmetrics\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'accuracy'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mD:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\learn_runner.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(experiment_fn, output_dir, schedule, run_config, hparams)\u001b[0m\n\u001b[1;32m    208\u001b[0m   \u001b[0mschedule\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mschedule\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0m_get_default_schedule\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrun_config\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    209\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m--> 210\u001b[0;31m   \u001b[1;32mreturn\u001b[0m \u001b[0m_execute_schedule\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mexperiment\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mschedule\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    211\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    212\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mD:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\learn_runner.py\u001b[0m in \u001b[0;36m_execute_schedule\u001b[0;34m(experiment, schedule)\u001b[0m\n\u001b[1;32m     45\u001b[0m     \u001b[0mlogging\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0merror\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'Allowed values for this experiment are: %s'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvalid_tasks\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m     46\u001b[0m     \u001b[1;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'Schedule references non-callable member %s'\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0mschedule\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m---> 47\u001b[0;31m   \u001b[1;32mreturn\u001b[0m \u001b[0mtask\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     48\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m     49\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mD:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\experiment.py\u001b[0m in \u001b[0;36mtrain_and_evaluate\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    500\u001b[0m                                       \u001b[0mname\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0meval_dir_suffix\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    501\u001b[0m                                       hooks=self._eval_hooks)\n\u001b[0;32m--> 502\u001b[0;31m     \u001b[0mexport_results\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_maybe_export\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0meval_result\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    503\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0meval_result\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mexport_results\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    504\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mD:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\experiment.py\u001b[0m in \u001b[0;36m_maybe_export\u001b[0;34m(self, eval_result, checkpoint_path)\u001b[0m\n\u001b[1;32m    595\u001b[0m                   compat.as_bytes(strategy.name)),\n\u001b[1;32m    596\u001b[0m               \u001b[0mcheckpoint_path\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mcheckpoint_path\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m--> 597\u001b[0;31m               eval_result=eval_result))\n\u001b[0m\u001b[1;32m    598\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    599\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0mexport_results\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mD:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\export_strategy.py\u001b[0m in \u001b[0;36mexport\u001b[0;34m(self, estimator, export_path, checkpoint_path, eval_result)\u001b[0m\n\u001b[1;32m     85\u001b[0m       \u001b[0mkwargs\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'eval_result'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0meval_result\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m     86\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m---> 87\u001b[0;31m     \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mexport_fn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mestimator\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mexport_path\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32mD:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\utils\\saved_model_export_utils.py\u001b[0m in \u001b[0;36mexport_fn\u001b[0;34m(estimator, export_dir_base, checkpoint_path)\u001b[0m\n\u001b[1;32m    410\u001b[0m           \u001b[0massets_extra\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0massets_extra\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    411\u001b[0m           \u001b[0mas_text\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mas_text\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m--> 412\u001b[0;31m           checkpoint_path=checkpoint_path)\n\u001b[0m\u001b[1;32m    413\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    414\u001b[0m     \u001b[0mgarbage_collect_exports\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mexport_dir_base\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mexports_to_keep\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mD:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\estimators\\estimator.py\u001b[0m in \u001b[0;36mexport_savedmodel\u001b[0;34m(self, export_dir_base, serving_input_fn, default_output_alternative_key, assets_extra, as_text, checkpoint_path)\u001b[0m\n\u001b[1;32m   1303\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m   1304\u001b[0m         \u001b[1;31m# Perform the export\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1305\u001b[0;31m         \u001b[0mbuilder\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0msaved_model_builder\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSavedModelBuilder\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mexport_dir\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1306\u001b[0m         builder.add_meta_graph_and_variables(\n\u001b[1;32m   1307\u001b[0m             \u001b[0msession\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mtag_constants\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSERVING\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mD:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\python\\saved_model\\builder_impl.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, export_dir)\u001b[0m\n\u001b[1;32m     88\u001b[0m           \"directory: %s\" % export_dir)\n\u001b[1;32m     89\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m---> 90\u001b[0;31m     \u001b[0mfile_io\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrecursive_create_dir\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_export_dir\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     91\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m     92\u001b[0m     \u001b[1;31m# Boolean to track whether variables and assets corresponding to the\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mD:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\python\\lib\\io\\file_io.py\u001b[0m in \u001b[0;36mrecursive_create_dir\u001b[0;34m(dirname)\u001b[0m\n\u001b[1;32m    365\u001b[0m   \"\"\"\n\u001b[1;32m    366\u001b[0m   \u001b[1;32mwith\u001b[0m \u001b[0merrors\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mraise_exception_on_not_ok_status\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mstatus\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m--> 367\u001b[0;31m     \u001b[0mpywrap_tensorflow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mRecursivelyCreateDir\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mcompat\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mas_bytes\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdirname\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstatus\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    368\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    369\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mD:\\AnnaConda\\AnnaConda3\\lib\\contextlib.py\u001b[0m in \u001b[0;36m__exit__\u001b[0;34m(self, type, value, traceback)\u001b[0m\n\u001b[1;32m     87\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mtype\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m     88\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m---> 89\u001b[0;31m                 \u001b[0mnext\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgen\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     90\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m     91\u001b[0m                 \u001b[1;32mreturn\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mD:\\AnnaConda\\AnnaConda3\\lib\\site-packages\\tensorflow\\python\\framework\\errors_impl.py\u001b[0m in \u001b[0;36mraise_exception_on_not_ok_status\u001b[0;34m()\u001b[0m\n\u001b[1;32m    464\u001b[0m           \u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    465\u001b[0m           \u001b[0mcompat\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpywrap_tensorflow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTF_Message\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mstatus\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m--> 466\u001b[0;31m           pywrap_tensorflow.TF_GetCode(status))\n\u001b[0m\u001b[1;32m    467\u001b[0m   \u001b[1;32mfinally\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m    468\u001b[0m     \u001b[0mpywrap_tensorflow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mTF_DeleteStatus\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mstatus\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNotFoundError\u001b[0m: Failed to create a directory: models/model_WIDE_AND_DEEP_1556720127\\export\\Servo\\1556720345"
     ]
    }
   ],
   "source": [
    "# Run the experiment\n",
    "\n",
    "model_dir=create_model_dir(model_type=MODEL_TYPE)\n",
    "metrics, output_folder = learn_runner.run(experiment_fn, model_dir)\n",
    "\n",
    "print('Accuracy: {}'.format(metrics['accuracy']))\n",
    "print('Model exported to {}'.format(output_folder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusions\n",
    "\n",
    "In this Juypter notebook, we have configured, created, and evaluated a Wide & Deep machine learning model, that combines the powers of a Linear Classifier with a Deep Neural Network, using TensorFlow's Estimator and Experiment classes.\n",
    "\n",
    "With this working example in your toolbelt, you are ready to explore the wide (and deep) world of machine learning with TensorFlow! Some ideas to help you get going:\n",
    "* Change the features we used today. Which columns do you think are correlated and should be crossed? Which ones do you think are just adding noise and could be removed to clean up the model?\n",
    "* Swap in an entirely new dataset! There are many datasets available on the web, or use a dataset you possess! Check out https://archive.ics.uci.edu/ml to find your own dataset. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
